Visual Studio Update 3 adds some new warnings for precompiled header generation. One of these, C4598, will prevent precompiled header builds of framework code due to our warnings as errors policy.
A fix for this issue is to add the new warning to the list of disabled warnings in the project file. The easiest way to do this is to do a search and replace across your source tree for *.vcxproj files and replace ";" with "98;".
I need to work out what the implications of this
are on how our precompiled headers are generated.
Version 6.7 of The Server Framework was released today.
This release is mainly a code simplification release which removes support for legacy compilers and operating systems. See
for more details. However, there are some breaking changes where smart buffers have replaced buffer references and this causes function signature changes. In addition there has been considerable churn in the Streaming Media Option Pack with knock on changes in the HTTP library code which needed to be made more complete to deal with serving HLS streams.
As always, see the release notes , for full details of all changes.
Breaking changes:
Breaking change JetByteTools::IO::CAsyncFileReader, JetByteTools::IO:: CAsyncFileWriter, JetByteTools::IO::CAsyncFileWriterEx and JetByteTools::IO::IAsyncIOStream now work in terms of JetByteTools::IO:CSmartBuffer where possible. This potentially reduces the reference counting activity on the buffers.
Breaking change Where possible JetByteTools::IO::CSmartBuffer has replaced raw pointers and references to buffers. This massively reduces the need to reference count buffers during normal I/O operations and increases performance, especially on NUMA architectures.
Bug fixes:
Bug fixes to JetByteTools::IO::CBuffer around the usage of m_maxBytesToRead.
Bug fix to JetByteTools::Win32::CCallbackTimerQueueEx::BeginTimeoutHandling() to prevent incrementing
m_nextTimeoutHanlde causing the value to wrap to InvalidTimeoutHandleValue which was
possible, but unlikely.
Dropped support for Visual Studio 2005 and Visual Studio 2008.
Dropped support for Windows XP.
Removed JETBYTE_HAS_INTERLOCKED_64 as it's true on all supported platforms now.
Removed JETBYTE_WARN_ON_SOCKADDR_STORAGE_DEF and JETBYTE_WARN_ON_WSPIAPI_COUNTOF_DEF as these are no longer relevant.
Removed JETBYTE_PERF_STREAM_SOCKETS_SKIP_EVENT_ON_HANDLE, JETBYTE_PERF_DATAGRAM_SOCKETS_SKIP_EVENT_ON_HANDLE and JETBYTE_PERF_FILE_WRITER_SKIP_EVENT_ON_HANDLE as these are always enabled now.
Removed JETBYTE_PERF_STREAM_SOCKETS_SKIP_MARSHAL_TO_IO_POOL and JETBYTE_PERF_DATAGRAM_SOCKETS_SKIP_MARSHAL_TO_IO_POOL as these are always enabled now. Marshalling was only required to work around Windows XP's I/O cancellation on thread termination policy.
Removed JETBYTE_DEPRECATE_SHARED_CRITICAL_SECTIONS, JETBYTE_DEPRECATE_SHARED_LOCK_SOCKETS, JETBYTE_DEPRECATE_CRITICAL_SECTION_2, JETBYTE_DEPRECATE_LOW_CONTENTION_BUFFER_ALLOCATOR, and JETBYTE_DEPRECATE_TLS_BUFFER_ALLOCATOR as these features are no longer available.
Removed SecureCRT.h as it's no longer required. It was used to map differences between the Visual Studio 2005 CRT and the "new" secure CRT.
Added the macro, SuppressLNK4221Warning(),
which can be put into a file in order suppress the MS Visual C++ Linker warning 4221 - "warning LNK4221: no
archive member will be inaccessible"
Turn off JETBYTE_INTRUSIVE_RED_BLACK_TREE_INTERNAL_STATE_FAILURE_EXCEPTIONS by default.
Removed JetByteTools::IO::CLowContentionBufferAllocator and JetByteTools::IO::CTLSBufferAllocator.
Added JetByteTools::IO::CBufferBasedBufferAlloctor which is a simple shim to allow an instance of JetByteTools::IO::IBuffer to be used as an implementation of JetByteTools::IO::IAllocateBuffers.
Removed JetByteTools::IO::IIOPool::DispatchToAll() as it was only required for issuing CancelIO() calls to all I/O threads on XP where CancelIOEx() wasn't available.
Added an overload of JetByteTools::IO::CNonPooledBuffer::Create() that takes a bufferSize and a pointer to data and a data length so that you can create a buffer that initially contains some data but that is larger than that data.
Added lots of standard buffer functionality to JetByteTools::IO::CNonPooledBuffer before deprecating it in favour of normal buffers obtained via JetByteTools::IO::IAllocateBuffers::AllocateCustomSizedBuffer().
Removed JetByteTools::IO::CAsyncFileWriter::ExecuteWritesOnCallingThreadIfSafe as it's the same as JetByteTools::IO::CAsyncFileWriter::ExecuteWritesOnCallingThread now that we no longer support Windows XP.
Added JetByteTools::IO::IBuffer::GetTotalLength() which returns the length of a set of buffers defined using an array of WSABUF structures.
Added JetByteTools::IO::CBufferChain::CopyBufferChain() which uses an instance of JetByteTools::IO::CBufferChain::IAllocateBufferHandles to create a duplicate of a given buffer chain where the duplicate contains handles to the buffers in the original chain.
Removed all of the code that was required to marshall I/O operations to the I/O threads on Windows XP.
Removed the concept of write sequencing. There's no need now that the marshalling code has been removed.
Removed the concept of "shared lock sockets" and, correspondingly "unique lock sockets". All socket objects now have their own lock.
Removed the option of setting a socket's lock's spinCount in the constructor of the socket allocator
Replaced some usages of JetByteTools::Win32::CCriticalSection with JetByteTools::Win32::CLockableObject
Removed JetByteTools::Win32::ICriticalSectionFactory, JetByteTools::Win32::ISharedCriticalSection, JetByteTools::Win32::IManageSharedCriticalSections, JetByteTools::Win32::CCriticalSection2, JetByteTools::Win32::CSharedCriticalSection, JetByteTools::Win32::CSharedCriticalSectionFactory, JetByteTools::Win32::CSmartSharedCriticalSection and JetByteTools::Win32::CUniqueCriticalSectionFactory. The concept of shared critical sections is no longer supported.
Removed JetByteTools::Win32::CThreadedCallbackTimerQueue::HybridTickCount64 and
JetByteTools::Win32::CThreadedCallbackTimerQueue::HybridTickCount64NoLock as the hybrid GetTickCount64()
implementation is no longer required as all supported platforms now provide GetTickCount64.
Removed JetByteTools::Win32::CCallbackTimerQueue. JetByteTools::Win32::CCallbackTimerQueueEx is now the only
timer queue implementation.
Removed JETBYTE_USE_CAPTURE_STACK_BACK_TRACE we now ALWAYS used CaptureStackBackTrace() so there's no need to make it optional.
Added new overloads for JetByteTools::Win32::GetFileNameFromPathName() and JetByteTools::Win32::StripFileNameFromPathName() which takes the path separator. This allows the functions to be used for file system paths or URI paths.
Added new overloads for JetByteTools::Win32::GetFileVersion() and JetByteTools::Win32::GetFileVersionString() which take languge IDs and charset IDs
Added JetByteTools::Win32::RemoveDirectoryContents().
Added JetByteTools::Win32::GetFileSize().
Added some code to the top and bottom of Utils.h which try to deal with situations where min and max have been defined as macros. We use the std::min() and std::max() template versions and macros confuse matters so the new code attempts to undef the macros if present and then redefine them at the end of the header.
Newly deprecated code:
Deprecated JetByteTools::IO::IAllocateMultiBufferHandles and the concept of "multi buffer handles".
The stacking of connection filters that can generate their own writes has been deprecated.
Compressing and deflating socket filters are now deprecated.
I hinted at the end of
that the 6.7 release might increase performance a little. Well, whilst the bulk of the changes in 6.7 are purely code cleaning and the removal of legacy support there is a fairly major functional change as well.
In most situations references or pointers to I/O buffers have been replaced with smart pointers. This change may cause some issues during an upgrade as you need to change some function signatures from IBuffer refs to CSmartBuffers. The advantage is that in many servers there will no longer be any need for buffer reference counting during normal I/O operations.
relies on reference counting to keep the objects that are used during the asynchronous operations alive until those operations complete. So we increment a counter on the socket object and also on the buffer object when we initiate an operation and then decrement the counters when the operation completes. I'm sure there are other ways to manage the object lifetime but this has worked well for us. The problem is that these increments, although they look like cheap operations, can be quite expensive, especially on NUMA hardware.
Whilst there's not much we can do about the reference count on the socket object, the buffer doesn't really need to be reference counted most of the time. Or, more's the point. The initial reference can (and should) be passed along rather than each stage taking and releasing its own reference. With a buffer you generally only want to be accessing it from one thread at a time and so you allocate it and then issue an operation and pass the reference you have off to the operation. When the operation completes the code captures the reference and takes ownership of it and then the buffer can be processed. If you're lucky you can then use the same buffer for a response to the operation and pass it back to the framework again.
This requires a few changes to your code but it's fairly straight forward. Your OnReadCompleted() handler will give you a CSmartBuffer and if you want to retain ownership of it after the handler returns then you simply need to detach the buffer from the CSmartBuffer you were given.
This is only "potentially faster" as it really depends on the structure of your server and how you deal with our I/O buffers but the framework is no longer standing in the way of this kind of optimisation, and we've removed a couple of reference adjustments in the normal operation flow.
We've only just shipped Release 6.6.5 of
but we already have another release that's just about to ship. This isn't because some horrible bug has slipped through our testing, it's because we've been planning to produce a 'clean up' release for some time. 6.7 is that release.
Lets be straight here, 6.7 is a release for us more than for you. The aim is to simplify our build/test and release process, remove dead code whilst introducing no new bugs and removing no functionality that you rely on.
So what does 6.7 achieve. Well, for a start we drop support for Visual Studio 2005 and 2008 and also for Windows XP. Removing support for these legacy compilers and operating systems means that we can remove all the code that was required just to support them. This massively simplifies our code base without removing anything that the code actually relies on to run on modern operating systems.
Windows Vista introduced
and we have supported these changes for a long time (over 8 years!). The code required to jump through hoops to make code running on Windows XP behave was complex. For example, Windows XP would cancel outstanding I/O requests if the thread that issued them exited before the I/O request completed. We had a marshalling system in place to ensure that I/O operations were only ever executed on threads that we controlled so that you'd never be faced with unexpectedly cancelled operations. All of that can go now.
Removing XP also means we no longer need to maintain an XP machine in our build farm. It's one less configuration that needs to be built and tested before a release.
Dropping support for VS2005 and 2008 removes 4 complete sets of builds (x86 and x64 for each compiler) plus all of the conditional code that was required to support the older compilers. At last we can start moving towards a slightly more modern C++, perhaps.
Some old c there's no need, on modern operating systems, to share locks. This worked really well back in the day, but, well, we were running on Windows NT at the time and resources were much more limited than they are now. All of the "Shared Critical Section" code is now gone. This has knock on effects into the Socket Tools library where all of the shared lock socket code has been removed. Nobody should be using that in 2016 anyway! You can no longer set a critical section's spin count in the socket allocator, it never really worked anyway as the lock was used for too many different things.
Some experimental code h The TLS and Low Contention buffer allocators are gone. The horrible "dispatch to all threads" cludge has been removed from the I/O pools (it was only there to support pre-Vista CancelIO() calls which are no longer needed now that we have CancelIOEx()).
The original callback timer queue that was based on GetTickCount() and which spawned
has gone. There's no need for the complexity when all supported operating systems have GetTickCount64().
Finally we've slimmed down our set of example servers. Removing servers which didn't add much value or which duplicated other examples. Again, this speeds our release process by speeding up the build and test stage as there are fewer servers to build and fewer tests to run.
So, what's in it for you? Well, a faster build/test/release cycle so new functionality and bug fixes can be released quicker and potentially faster code in some circumstances. There's no great rush to upgrade if you don't want to, but we'll be focusing on the 6.7 code base going forwards.
Version 6.6.5 of The Server Framework was released today.
This release is mainly a feature release with a few bug fixes.
As always, see the release notes , for full details of all changes.
Bug fixes:
Bug fix to JetByteTools::Socket::TAsyncSocket::ProcessAndGetNextOperation(). We now wrap the body of the function in an exception handler and abort the connection with JetByteTools::Socket::ConnectionClosureReason::FatalErrorAbort if an exception is thrown during processing. This fixes a bug whereby the connection would otherwise hang in these situations.
Bug fix to JetByteTools::Win32::TReentrantLockableObjectTracksLockingThread to add a cast which is needed for some compilers.
Bug fix to JetByteTools::WebSocket::HyBi::CProtocolHandler::HandleData() to remove an incorrect internal state validation exception which would generate spurious "No space left in read buffer." exceptions.
Added JetByteTools::IO::IBuffer::OnBufferAddedToPool(), JetByteTools::IO::IBuffer::OnBufferRemovedFromPool(), JetByteTools::IO::IManageBufferLifeCycle::OnBufferAddedToPool() and JetByteTools::IO::IManageBufferLifeCycle::OnBufferRemovedFromPool(). These allow for correct management of dynamically allocated buffer data. Previously JetByteTools::IO::IMonitorBufferAllocation::OnBufferAllocated() was called whenever a buffer was allocated from the allocator OR a custom pool and JetByteTools::IO::IMonitorBufferAllocation::OnBufferReleased() was only called when the buffer was released to the allocator. This made it impossible to manage dynamically allocated buffer data that was created and destroyed by a derived allocator object using the monitoring interface to monitor the underlying allocator. Now JetByteTools::IO::IMonitorBufferAllocation::OnBufferReleased() is called when the buffer is added to a custom pool and this allows the monitor to match allocations and releases exactly.
Added JetByteTools::IO::IAllocateBuffers::AllocateCustomSizedBuffer() which allows you to allocate a custom buffer from any allocator. The buffer will have the same user data slots as any other buffer allocated from the allocator but if it is larger than the allocator's buffer size the new buffer will NOT be pooled upon release.
Added JETBYTE_ILLEGAL_BUFFER_USER_DATA_EXCEPTIONS which defaults to 0 and when set turns on range checking for user data indices in JetByteTools::IO::CBuffer and an index out of range exception is thrown if necessary.
Added the ability to pass JUST a custom allocator into the constructor of JetByteTools::IO::CRotatingAsyncFileLog and allow the file log to manage the other resources that it needs directly. Previously you had to provide all of the resources or none.
Added an override for JetByteTools::Socket::ISocketCallback::OnError() which takes a DWORD error code so that client code can selectively ignore errors by error code. Previously the only way to ignore errors was by the error message itself which is localised and therefore impossible to match reliably.
Rationalised the status changes for JetByteTools::Socket::TStreamSocketServer&& and JetByteTools::Socket::TStreamSocketServerEx&& to remove some strangeness when shutdowns are initiated after the server has already shut down.
Added new value to JetByteTools::Socket::ConnectionClosureReason, FatalErrorAbort. This is used if the framework itself needs to abort a connection for any reason.
Added JetByteTools::Socket::CStreamSocketNamedConnectionCollection::GetConnectionName() which returns the name of a given connection.
Added JetByteTools::Socket::CStreamSocketBroadcastableConnectionCollection::BroadcastToAllExcept() which broadcasts a buffer to all connections except the supplied connection.
Added JetByteTools::Socket::StreamSocketBroadcastableNamedConnectionCollection.
Added JETBYTE_TRACK_ADDRESS_REFERENCES which defaults to 0 and when set enables reference tracking of JetByteTools::Socket::CAddressImpl objects in the same way that tracking can be enabled for sockets and buffers.
Added JETBYTE_STREAM_SOCKETS_DISPATCH_OTHER_SOCKETS_DURING_COMPLETION_HANDLING and JETBYTE_DATAGRAM_SOCKETS_DISPATCH_OTHER_SOCKETS_DURING_COMPLETION_HANDLING which both default to 0 and this changes the default behaviour from previous releases. These control how we deal with dispatching operations from other sockets whilst dispatching events from a socket... This only affects designs where one connection can write to another connection. In such a design, in earlier versions, we would allow processing of operations from the 'other' connection whilst we're processing operations from the main connection. For example. If we're in the read completion handler for a connection and we issue a read on another connection and that read completes immediately, inline, then we would begin to handle the completion and end up in the read completion handler for the other socket. This could cause issues with lock inversions if each connection needs to take out locks. The new default is to NOT handle the inline completions for the 'other' socket inline but instead to queue them until after the current socket's operation handler completes. You can revert to the old behaviour by setting these to 1 in your Config.h file.
Added JETBYTE_DUMP_NAMED_INDEX_DETAILS_ON_LOCK which defaults to 0 and when set causes the names of named indices to be dumped to the default debug trace log when the indices are locked for the first time. This can be useful in tracking down mismatches between different named index providers.
Changed JetByteTools::Win32::ICreateMiniDumps::GenerateDumpFileName() so that it takes a 'type' which, if it's not an empty string, is added into the generated name just after the filename base portion. This allows you to group dumps by type - for situations where a you are generating dumps for different reasons.
Changed JetByteTools::Win32::CMiniDumper to take into account the changes to JetByteTools::Win32::ICreateMiniDumps::GenerateDumpFileName(). This means changing JetByteTools::Win32::CMiniDumper::CreateMiniDump(), JetByteTools::Win32::CMiniDumper::CreateMaxiDump() and JetByteTools::Win32::CMiniDumper::CreateFullDump() so that they take the 'type' name and adding JetByteTools::Win32::CMiniDumper::CreateMiniDumpWithFileName(), JetByteTools::Win32::CMiniDumper::CreateMaxiDumpWithFileName() and JetByteTools::Win32::CMiniDumper::CreateFullDumpWithFileName() to disambiguate certain call signatures.
Changed JetByteTools::Win32::CMiniDumpGenerator to take into account the changes to JetByteTools::Win32::ICreateMiniDumps::GenerateDumpFileName(). This means changing JetByteTools::Win32::CMiniDumpGeneratorGenerateDumpHere() and JetByteTools::Win32::CMiniDumpGenerator::GenerateDump() to take the new 'type' name and changing the bool used allow calls to ignore dump limits to an enum.
Added the concept of 'per type' mini dump limits. These can be set through JetByteTools::Win32::CMiniDumpGenerator::SetMaxDumps() by specifying a type string to set the maximums for. If a per type limit is not set then the global limit is used.
Added a new enum value to the JetByteTools::Win32::CMiniDumpGenerator::MaxDumpLimits enum to enable the use of 'per type' limits.
Changed JetByteTools::Win32::CGlobalErrorHandler to take into advantages of the dump 'type' changes to categorise the types of dumps produced.
Cosmetic changes to JetByteTools::Win32::CThread to adjust the optional thread name tracking structure.
Added JetByteTools::Win32::CPerThreadErrorLog which can be instantiated on a thread to install handlers for std::terminate and std::unexpected. These handlers are per thread. Note that you should still use JetByteTools::Win32::CGlobalErrorHandler on the main thread as this also installs some process-wide handlers.
JetByteTools::Win32::CGlobalErrorHandler now derives from JetByteTools::Win32::CPerThreadErrorLog.
The signature for the constructor of JetByteTools::Win32::CSEHException::Translator has been changed to take an int which is not used for anything except to allow you to locate 'old' instances of the class and update them to instances of JetByteTools::Win32::CPerThreadErrorLog which include the functionality of the exception translator. This change can be turned off by defining JETBYTE_BREAK_SEH_EXCEPTION_TRANSLATOR_COMPATABILITY to 0 in config.h.
Added JETBYTE_INSTALL_PER_THREAD_ERROR_HANDLER_IN_CTHREAD which defaults to 1 and when set installs a JetByteTools::Win32::CPerThreadErrorLog object onto all JetByteTools::Win32::CThread object's threads.
All threads created by framework code will install an instance of JetByteTools::Win32::CPerThreadErrorLog whether or not JETBYTE_INSTALL_PER_THREAD_ERROR_HANDLER_IN_CTHREAD is defined as 0.
Mini dumps created by JetByteTools::Win32::CCrtReportHook are now always created even if any program limits on the number of dumps to create has been met or exceeded.
Added JetByteTools::Win32::IIOCPWorkerThreadCallback::ProcessEx() which can be overridden to do the work that JetByteTools::Win32::IIOCPWorkerThreadCallback::Process() would do and then return a bool to indicate if you need to be informed when the queue has no more items to process. The default implementation simply calls JetByteTools::Win32::IIOCPWorkerThreadCallback::Process() and returns false.
Added JetByteTools::Win32::IIOCPWorkerThreadCallback::NoItemsToProcess() which is called once when the queue becomes empty if the last call to JetByteTools::Win32::IIOCPWorkerThreadCallback::ProcessEx() or JetByteTools::Win32::IIOCPWorkerThreadCallback::NoItemsToProcess() returned true. The default implementation simply returns false. Note that returning true will cause the queue to be checked again and if still empty for JetByteTools::Win32::IIOCPWorkerThreadCallback::NoItemsToProcess() to be called again.
Adjusted how services shut down in error situations when the service has failure actions set. When failure actions are enabled for a service we will now shut down in such a way that they are triggered. This involves terminating the thread that called StartServiceCtrlDispatcher() and making sure that we don't call SetServiceStatus() with a status of SERVICE_STOPPED. We shut down normally if the service is in the process or starting up or shutting down when the error occurs so that we avoid the SCM repeatedly restarting a service that we can't shut down. If "Enable actions for stops with errors" is set then the SCM will restart the service if it returns a non-zero exit code, even if it calls SetServiceStatus() with a status of SERVICE_STOPPED and there's no way for us to prevent it restarting a service that is failing during init or shutdown.
JetByteTools::WebSocket::CHeaders can now handle adding headers with no value.
A while back a client of mine had an issue with a TLS 1.2 connection failing during the TLS handshake. We couldn't see any issues with the code and if we only enabled TLS 1.1 on the server then the connection handshake worked just fine.
Eventually we tracked the issue down to the fact that the certificate in use had been signed with MD5 and that MD5 isn't a valid hash algorithm for TLS 1.2 and so the handshake failed when SChannel attempted to validate the certificate and found that it was unacceptable. There's a Microsoft blog posting about this problem .
Annoyingly the error message for this is: "The client and server cannot communicate, because they do not possess a common algorithm." which is bang on the money, but doesn't help much until after you've worked out what the problem is. We were already dumping out the enabled algorithms and protocols and couldn't understand why the connection either wasn't working or wasn't downgrading to TLS 1.1.
I've now added some certificate investigation code to the point where my example SChannel servers set up their contexts. This at least allows me to warn and degrade to TLS 1.1 if the certificate is not going to work with TLS 1.2. In production systems I expect we'd just fail to start up.
CCredentials::D
data.cCreds = 1;
data.paCred = &pCertC
// Note that for TLS 1.2 you need a certificate that is signed with SHA1 and NOT MD5.
// If you have an MD5 certificate then TLS 1.2 will not connect and will NOT downgrade
// to TLS 1.1 or something else...
DWORD dataSize = 0;
if (::CertGetCertificateContextProperty(
pCertContext,
CERT_SIGN_HASH_CNG_ALG_PROP_ID,
&dataSize))
ByteBuffer buffer(dataSize);
BYTE *pData = buffer.GetBuffer();
if (::CertGetCertificateContextProperty(
pCertContext,
CERT_SIGN_HASH_CNG_ALG_PROP_ID,
&dataSize))
const _tstring signHashDetails(reinterpret_cast&tchar *&(pData), dataSize);
if (signHashDetails.find(_T("MD5")) != _tstring::npos)
if (enabledProtocols == 0 || enabledProtocols & SP_PROT_TLS1_2)
OutputEx(_T("Certificate is signed with: ") + signHashDetails);
OutputEx(_T("MD5 hashes do not work with TLS 1.2 and connection downgrade does NOT occur"));
OutputEx(_T("This results in failure to connect. Disabling TLS 1.2"));
if (enabledProtocols)
enabledProtocols &= ~SP_PROT_TLS1_2;
if (!enabledProtocols)
// if enabledProtocols is zero then all protocols are
// enabled, so we need to explicitly enable everything
// except TLS1.2
enabledProtocols = SP_PROT_SSL3TLS1;
data.grbitEnabledProtocols = enabledP
Version 6.6.4 of The Server Framework was released today.
This release is mainly a bug fix release for clients using WebSockets over SSL.
As always, see the release notes , for full details of all changes.
Bug fixes:
Bug fix to JetByteTools::Win32::CallbackTimerQueueBase which prevents the timeout handle from ever being incremented to zero. It's unlikely but possible.
Bug fix to JetByteTools::Win32::CBuffer and JetByteTools::Win32::CLockFreeBuffer to make code take notice of when m_maxBytesToRead is set. This was causing issues with WebSockets over SSL where the WebSocket code was setting a max read value and the SSL connector was ignoring it and providing more data than it should.
Bug fix to JetByteTools::SSPI::SChannel::CAsyncConnector::PerformWrite() so that we deal with exceptions appropriately.
Bug fix to JetByteTools::Socket::CFilterDataBase::WriteData() so that we deal with exceptions appropriately.
Bug fix for a highly unlikely possible situation in JetByteTools::WebSocket::HyBi::CProtocolHandler::HandleData() where a buffer in the read request buffers queue contains no space for more data.
Adjusted all includes to remove relative paths and instead be absolute in relation to the library root. So, rather than #include "..\Blah.h" it would always be #include "JetByteTools\Lib\Blah.h" this is due to a new warning in Visual Studio 2015 update 1.
Added JetByteTools::Socket::ListenBacklogMaxwhich is a JetByteTools::Socket::ListenBacklog value that can be used to cause the listen queue for a socket to be set to SOMAXCONN.
Added JetByteTools::Socket::TStreamSocketServerEx::SoMaxNumPendingAccepts which is the number of accepts that are posted if a listen backlog of JetByteTools::Socket::ListenBacklogMax is specified for the number of pending accepts. At present this is set to 200.
Added JetByteTools::Socket::TStreamSocketServerEx::SetListeningParameters() which can be called after construction but before starting to accept connections and can specify the listen backlog and number of receives to post separately. This allows you to set the listen backlog to SOMAXCONN using JetByteTools::Socket::ListenBacklogMax and set the number of receives to something other than JetByteTools::Socket::TStreamSocketServerEx::SoMaxNumPendingAccepts. You can also set a flag which tells the server not to close the listening socket when the server reaches a connection limit, instead it will simply stop posting accepts and so new connections will get to wait in the listen queue until they time out or until the server posts a new accept request.
Bug fix to to JetByteTools::Socket::TStreamSocketServerEx so that OnMaxConnections() is called correctly when the connection limit is reached. Previously it would often only be called the first time that the limit was reached.
Version 6.6.3 of The Server Framework was released today.
This release is mainly a bug fix release but it also adds support for Visual Studio 2015 and Windows 10 (though we don't explicitly use any Windows 10 APIs). There are quite a lot of small changes due to us running Gimpel Lint over the code. Most of the changes will have fixed potential issues rather than issues that have actually been reported.
However, there are two serious fixes for
for an issue that was introduced in Release 6.6.2 and which could cause corruption of the data flow in some situations and stalling and CPU pegging in another. There is also a fix which may prevent UDP datagram corruption, See
for more details.
As always, see the release notes , for full details of all changes.
All clients using Release 6.6.2 of are advised to upgrade to this release.
Bug fixes:
JetByteTools::IO::CBuffer::Clear() which now correctly sets m_ioSize to zero.
JetByteTools::IO::CBufferList::BufferData so that it cannot access pBuffer before it's initialised.
JetByteTools::IO::CLockableBufferProcessor::~CLockableBufferProcessor() to remove potential buffer leak.
JetByteTools::IO::CAsyncFileReader where we were checking the validity of data response flags incorrectly.
JetByteTools::SSPI::SChannel::CAsyncConnector::PerformWrite() where we could sometimes send invalid data.
JetByteTools::SSPI::SChannel::CAsyncConnector where we could sometimes get stuck in an infinite loop if we had decrypted data available but no application level read pending.
JetByteTools::Service::CService::ServiceMain() so that we set and clear the m_serviceStarting flag correctly.
Fix to all WSASend() and WSARecv() calls to prevent multiple threads from calling into the API on the same socket at the same time. See
for more details.
JetByteTools::Win32::CCallbackTimerWheel::EndTimeoutHandling() where we were leaking one shot timers.
JetByteTools::Win32::CThreadPool::OnThreadPoolThreadStopped() and JetByteTools::Win32::CThreadPoolEx::OnThreadPoolThreadStopped() so that the call to the monitor's OnThreadPoolStopped() function is consistent and in the right place.
JetByteTools::Win32::CSmartHandle to deal with closing pseudo thread or process handles.
Fix to JetByteTools::Socket::CFilterDataBase to ensure that each filter knows how many writes it has issued and that they pass completions to the next layer when they should.
JetByteTools::Win32::TReusableIdManager::InternalTryFree() so that we now correctly merge intervals. Note that this bug was purely in the way we stored the intervals (which was less efficient than it could be) and did not affect the function of the manager.
JetByteTools::Win32::TZeroInitialiseExpandableBuffer::Resize() so that it doesn't trash memory if you reduce the size of the buffer.
JetByteTools::Win32::TZeroInitialiseExpandableBuffer a fix to the assignment operator so that it actually compiles and works.
Ran Visual Lint using Gimpel PC Lint on all code and adjusted to remove warnings and fix bugs.
Added support for Visual Studio 2015.
Removed all use of exception specifications. We only ever used throw() but that's now gone too.
Protected non-virtual destructors on interfaces are now virtual even though they you can't delete the object via the interface.
All destructors that could throw exceptions now have optional "log and swallow" exception handlers which are enabled by default. This is better than ignoring the problem and being faced with a call to std::terminate() which can be hard to track down.
Changed JetByteTools::IO::CNonPooledBuffer so that it doesn't have to be immutable. You can now create a non-pooled buffer and add data to it after creation.
Added JetByteTools::IO::CBufferChain::AvailableSpace() and JetByteTools::IO::CBufferChain::HasAvailableSpace()
Added JetByteTools::IO::CBufferChain::RemoveAsSingleBuffer() which will either consolidate a chain into the first buffer (if all the data will fit) or create a JetByteTools::IO::CNonPooledBuffer of the required size and return the data inthe new buffer.
Added the concept of a "purge handler" for buffer chains. This comes in the form of an interface JetByteTools::IO::CBufferChain::IHandlePurgedBufferRelease and a new override for JetByteTools::IO::CBufferChain::Purge which takes a purge handler and some user data. If the new version of purge is called the purge handler is called for each buffer purged from the chain and it is responsible for disposing of the purged buffer, normally by calling Release() on it. The reason for this is that it's now possible to have a buffer chain of buffers which are 'detached' buffer chains. When purge is called in this situation a purge handler can now be supplied which recreates the buffer chains and, in turn, purges them.
Added new constructors for JetByteTools::IO::CAsyncFileWriter and JetByteTools::IO::CAsyncFileWriterEx that take a filename and a JetByteTools::Win32::CSmartHandle to an open file.
Changed constructor for JetByteTools::IO::CAsyncFileWriter that previously took a HANDLE to take a JetByteTools::Win32::CSmartHandle
Added JetByteTools::IO::CBufferAllocator::GetNumActive() and JetByteTools::IO::CBufferAllocator::GetNumPooled()
Added explicit support for building against OpenSSL libs built with VS2015 as previous versions are no longer link compatible. Note that these #pragma link lines assume our internal naming convention for OpenSSL libs.
Added JetByteTools::IO::CBufferChain::Detach(), JetByteTools::IO::CBufferChain::Attach() and JetByteTools::IO::CBufferChain::Splice(). These allow you to take a buffer chain and 'detach' it from the buffer chain object so that you can work in terms of a single buffer (with the additional buffers chained on in the normal way but inaccessible). You can then take such a buffer and 'attach' it back to a buffer chain object. This allows you to pass a buffer chain through an interface which only understands single buffers. Splice() allows you to take an existing buffer chain and add a 'detached' buffer chain to the end of it.
Changed the number of tabs used in the JetByteTools::Service::CServiceManager::GetExtraHelp() message so that the remove option lines up correctly. I think this may be a dialog font issue and it may be inconsistent on different platforms.
Added JetByteTools::Win32::CActivatableObject::CommandWrapper to remove duplication in the users of JetByteTools::Win32::CActivatableObject
Removed JetByteTools::Win32::CFileChangeMonitor as it was not used or compiled. The functionality is contained in JetByteTools::Win32::CDirectoryChangeMonitor.
Added JetByteTools::Win32::CNTPTime which converts SYSTEMTIME to and from NTP timestamps.
Changes to the various JetByteTools::Win32::ToString() implementations so that they remain consistent across Visual Studio versions. Visual Studio 2015 changes how precision is used in the printf family of functions and provides greater default precision. We limit precision to 17 in JetByteTools::Win32::ToString() so that the output is consistent with earlier versions of Visual Studio.
JetByteTools::Win32::TReusableIdManager::Release() now returns the id.
JetByteTools::Win32::CCommandLine::Parse() now takes a reference to a string that can be used to return an error message.
Added JetByteTools::Win32::IQueueTimers::SetTimerWithRefCountedTimer(), JetByteTools::Win32::IQueueTimers::CancelTimerWithRefCountedTimer() and JetByteTools::Win32::IQueueTimers::DestroyTimerWithRefCountedTimer() which deal with the common pattern of code required when the timer handle is a reference counted class.
Added JetByteTools::Win32::IQueueTimers::RefCountedTimer.
Added JetByteTools::Win32::IManageEnvironmentVariables::TryDeleteVariable().
Added locking to JetByteTools::Win32::CEnvironmentBlock so that it's now safe to use from multiple threads.
Added checking when we lock JetByteTools::Win32::TLockableObjectPotentialOwner&& and JetByteTools::Win32::TReentrantLockableObjectPotentialOwner&& so that an exception is thrown if it's already locked. Also changed how JetByteTools::Win32::TReentrantLockableObjectPotentialOwner&& is implemented as it was unnecessarilly complex.
Breaking Change VS2015's exponent doesn't include a leading 0 when calling sprintf, etc. This changes the output for some calls to JetByteTools::Win32::ToString() and makes it inconsistent with all previous releases.
Last week I learnt something new, which is always good. Unfortunately it was that for over 15 years I'd been working under a misconception about how an API worked.
Tl;dr When using WSARecv() and WSASend() from multiple threads on a single socket you must ensure that only one thread calls into the API at a given time. Failure to do so can cause buffer corruption.
It all began with
on StackOverflow and I dived in and gave my usual response. Unfortunately my usual response was wrong and after spending some time talking to the person who had asked the question and running their test code I realised that I've been mistaken about this for a long time.
My view has always been that if you issue multiple WSARecv() calls on a single socket then you need to be aware that the completions may be handled out of sequence by the threads that you have servicing your I/O completion port. This is purely due to thread scheduling and the actual calls to WSARecv() are thread safe in themselves. I wrote about this
back in 2002. My belief was that you could safely call WSARecv() from multiple threads on the same socket at the same time and the only problem would be resequencing the reads once you processed them. Unfortunately this is incorrect as the example code attached to the question shows.
The example code is somewhat contrived for a TCP socket in that it doesn't care about the sequencing of the read completions and it doesn't care about processing the TCP stream out of sequence. It issues multiple WSARecv() calls from multiple threads and the data being sent is simply a series of bytes
where the next byte is the value of the current byte plus one and where we wrap back to zero after a 'max value' is reached.
Such a stream with a max value of 7 would look like this: 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x00, 0x01, 0x02, 0x03...
Validating such a TCP stream is as simple as taking any read that completes in any order and simply ensuring that the bytes that are contained in the buffer that has been returned follow the expected pattern. Starting from the first byte, whatever value it is, subsequent bytes must be one greater until the 'max value' is reached at which point the wrap to zero and continue to increment. Assuming my long held beliefs were true then it didn't matter how many threads were issuing WSARecv() calls on the socket at the same time, the resulting slices of the TCP stream should all be valid.
Unfortunately the test program fails to prove this and instead proves that without synchronisation around the WSARecv() call the returned stream slices can be corrupt. That is the data in the buffers returned from the read completion can include bytes that do not follow the expected pattern.
Of course the way that the test program uses the API is of limited use as I can't think of a TCP stream where it would be useful to be able to process the stream in randomly sized chunks with no need to have processed the data that comes before the current data. One of the reasons that I believed that I understood the requirements of the API was that I never used it this way. In systems where multiple reads on TCP streams were allowed I would always increment a sequence number, put the sequence number in the read buffer's meta data and then issue the WSARecv() all as an atomic step with locking around it. This made it possible to correctly sequence the read completions and also had the side effect of preventing multiple threads calling
WSARecv() on a single socket at the same time. However, with UDP sockets the broken usage pattern is much more likely and I think that I may have seen the corruption on a client's system in the past - we're testing with fixed code now.
I've yet to fully investigate the WSASend() side of the problem but I'm assuming that the issue is the same. The person who asked the question on StackOverflow has seen data corruption when the receive side is protected by synchronisation and the send side isn't and I've no reason to doubt his analysis. I would like to think that calling WSASend() on one thread and WSARecv(), on another on the same socket, at the same time, is OK, but for now I'm assuming it's not and simply using a single lock per socket around both calls.
The current
has this to say about thread safety: If you are using I/O completion ports, be aware that the order of calls made to WSARecv is also the order in which the buffers are populated. WSARecv should not be called on the same socket simultaneously from different threads, because it can result in an unpredictable buffer order. Which, IMHO, doesn't actually address this issue. It mentions the unpredictable buffer order, which is expected, but not the buffer corruption. Also
has an identical note with "WSARecv" replaced by "WSASend", which, when you think about it, doesn't actually make sense at all. I don't remember seeing these notes when I was first looking at the documentation, but who knows (yes, I'd like a fully preserved set of all revisions to the MSDN docs so that I can diff back to what the docs said when I) ). , doesn't mention any thread safety issues at all in its coverage of WSASend() and WSARecv() as far as I can see from a brief scan.
Apart from the one known case of UDP datagram corruption that may be caused by my misunderstanding I think most of our systems are pretty safe just by their design. However, there will be a fix included in the 6.6.3 release and 7.0 wont be susceptible to this problem at all due to its
usage at the socket level.
It's always good to learn something new, more so when it challenges long held beliefs and especially when it proves those beliefs to have been incorrect.
The 7.0 release of The Server Framework, which is likely to be released early next year, will no longer support Visual Studio 2005 or Visual Studio 2008.
The 6.6.x releases will be the last to support these compilers.
Please get in touch immediately if this will be a problem for you.
Follow us on Twitter:
I usually write about the development of The&Server&Framework, a super scalable, high performance, C++, I/O Completion Port based framework for writing servers and clients on Windows platforms.
Find recent content on the
or look in the
to find all content.