Multi-threaded command-line text search proof-of-concept application Win32 C++ (Nov 2021)
Budget: £250 – £750 GBP
We are looking for an experienced developer to produce a small command-line, multi-threaded, text search proof-of-concept (PoC) application. Depending on the quality of the code produced and the performance achieved, the project may then develop into a more fully featured GUI application.
Objective:
Achieve maximum search performance possible. Target >= 1GB/sec on modern hardware
Prerequisites:
1. Code to be written in C++ using VS 2017 or later using C++11 or later
2. Code designed to run on Windows 10 or later
3. Target file IO data must be via our unbuffered data access class with a binary ReadFile() style interface
4. Support compilation as either 32-bit or 64-bit application
5. Not use memory mapped file IO
6. Source code to uploaded to our private GitHub repo at each stage of development
7. Copyright in any solution to be ours on final payment
Specification:
1. Parse command-line for
either:
a. List of search terms e.g. Must support at least 20 terms
b. Text file containing search terms e.g. Must support at least 1,000 terms
NB: Search terms may include up to two * wildcards
and:
c. Optional argument indicating if search should for ANSI text, Unicode text or both
d. Optional argument indicating if match should be case-sensitive or case-insensitive
e. Optional argument indicating file offset to start at. e.g. default is 0
f. Optional argument indicating number of bytes to process. e.g. default is all
g. Optional argument indicating maximum number of threads
h. Optional argument indicating maximum length for wildcard match e.g. default is 10 characters
i. Target file name to search e.g. Expect a file that is >= 8GB in size
2. Report error if required command-line argument missing or invalid
3. Open target file using our unbuffered file access class
NB: This uses a ReadFile() style API to access to the data. There are several reasons for insisting on this class but the main reason is to ensure that performance measure is independent of normal OS caching. This class will be provided to the winning bidder. In the meantime, it is sufficient to say that it opens a file using CreateFile() and reads binary data using ReadFile() API calls
4. Scan file for text matching specified criteria and report offset of each found term
5. During operation, periodically report % of operation completed and current MB/sec performance
5. On completion, report final performance in MB/sec
Suggested implementation:
The key to achieving maximum performance is to make good use of threads. We would suggest that a good approach would be to allocate a number of threads equivalent to the number of logical cores available and then divide the search terms across each thread such that they are balanced. We assume that allocating more threads than cores available would be pointless. Suggested approach:
1. Allocate large search cache e.g. ~100MB
2. Fill cache using provided file IO class
3. Allocate pool of threads based on the number of logical cores
4. Allocate search terms across threads. e.g. Each thread would have a similar (balanced) number of terms
5. Set threads to work on search cache simultaneously looking for terms. e.g. If number of terms balanced then threads should complete in similar time
6. When all threads complete, read next chunk of data into search cache and repeat
Payment:
Payment will be made in two stages:
a. 50% on completion of working prototype that show good performance characteristics and supports essential features above (including source code)
b. 50% on completion of finished PoC application, our testing and bug fixes to fix issues and ensure specification is fully met (including source code)
Objective:
Achieve maximum search performance possible. Target >= 1GB/sec on modern hardware
Prerequisites:
1. Code to be written in C++ using VS 2017 or later using C++11 or later
2. Code designed to run on Windows 10 or later
3. Target file IO data must be via our unbuffered data access class with a binary ReadFile() style interface
4. Support compilation as either 32-bit or 64-bit application
5. Not use memory mapped file IO
6. Source code to uploaded to our private GitHub repo at each stage of development
7. Copyright in any solution to be ours on final payment
Specification:
1. Parse command-line for
either:
a. List of search terms e.g. Must support at least 20 terms
b. Text file containing search terms e.g. Must support at least 1,000 terms
NB: Search terms may include up to two * wildcards
and:
c. Optional argument indicating if search should for ANSI text, Unicode text or both
d. Optional argument indicating if match should be case-sensitive or case-insensitive
e. Optional argument indicating file offset to start at. e.g. default is 0
f. Optional argument indicating number of bytes to process. e.g. default is all
g. Optional argument indicating maximum number of threads
h. Optional argument indicating maximum length for wildcard match e.g. default is 10 characters
i. Target file name to search e.g. Expect a file that is >= 8GB in size
2. Report error if required command-line argument missing or invalid
3. Open target file using our unbuffered file access class
NB: This uses a ReadFile() style API to access to the data. There are several reasons for insisting on this class but the main reason is to ensure that performance measure is independent of normal OS caching. This class will be provided to the winning bidder. In the meantime, it is sufficient to say that it opens a file using CreateFile() and reads binary data using ReadFile() API calls
4. Scan file for text matching specified criteria and report offset of each found term
5. During operation, periodically report % of operation completed and current MB/sec performance
5. On completion, report final performance in MB/sec
Suggested implementation:
The key to achieving maximum performance is to make good use of threads. We would suggest that a good approach would be to allocate a number of threads equivalent to the number of logical cores available and then divide the search terms across each thread such that they are balanced. We assume that allocating more threads than cores available would be pointless. Suggested approach:
1. Allocate large search cache e.g. ~100MB
2. Fill cache using provided file IO class
3. Allocate pool of threads based on the number of logical cores
4. Allocate search terms across threads. e.g. Each thread would have a similar (balanced) number of terms
5. Set threads to work on search cache simultaneously looking for terms. e.g. If number of terms balanced then threads should complete in similar time
6. When all threads complete, read next chunk of data into search cache and repeat
Payment:
Payment will be made in two stages:
a. 50% on completion of working prototype that show good performance characteristics and supports essential features above (including source code)
b. 50% on completion of finished PoC application, our testing and bug fixes to fix issues and ensure specification is fully met (including source code)