Rate the quality of 14k images (academic study)
Budget: $2 – $8 USD
Rate the technical quality of photos using the provided Chrome-based app. The project is part of an academic study on the perception of image quality.
OVERVIEW
You will be using our browser-based app (IQAvi) to select the quality of 13,770 images. Use the slider below each image to choose a quality rating from Bad (1) to Excellent (100). Before participating in the study, please review the task instructions: https://bit.ly/iqavi-instr.
PARTICIPANT REQUIREMENTS
- experience with photography, design, or visual arts
- normal or corrected-to-normal vision
HARDWARE REQUIREMENTS
- Use a sufficiently large physical screen (15”+) with a resolution greater than 2500x1400 pixels. Do not use a smartphone or a tablet.
- Adjust your screen brightness and contrast to see each image clearly. It is possible to miss details if your screen brightness is too low relative to the ambient illumination.
- Use a fast and stable internet connection (5+ Mbps) to have little or no delay when loading images (the average image size is 2 Mb).
WORK PLAN
1. It is essential to read the task instructions (https://bit.ly/iqavi-instr) to clearly understand what technical quality is and how to use the rating scale provided.
2. A list of experiment links will be provided, which you will open one by one, from top to bottom, in your Chrome web browser. Each link (experiment session) contains a batch of 112 images.
3. Fill in your system information and enter your freelancer.com profile URL when prompted to identify your submission.
4. Click on “start experiment.”
5. Switch your browser to Full-Screen mode (bottom-right button in the IQAvi App) to maximize the visible area for large resolution images (important!)
6. Choose the quality for each image by dragging the slider at the bottom of the page.
7. When finished with a batch of images, click the submit button to get your completion code.
8. Copy and paste each completion code into the corresponding row of the provided CSV file.
9. Complete all the sessions (including training) and send the filled CSV file to me.
SUGGESTIONS
- In the training session (https://bit.ly/iqavi-10k-training), we will present you with 160 images. Half of them show the correct answer to train your understanding of the quality scale. The other half evaluates your proficiency in judging the quality of images. Use this time to practice. To continue with the main experiment, you will need to pass the training with an accuracy greater than 50% (ratio of correct to all answers). Taking the training session is not mandatory before taking up the project. Still, it is compulsory to continue with the complete experiment.
- Providing good quality annotations requires paying attention to details. Please spend sufficient time per image. It is sometimes necessary to spend up to 10 seconds to explore/pan an image to find defects that affect its quality; however, the average time per image is around 3 seconds.
- Test images are used to assess your level of attention to the task and the accuracy of your answers. Please carefully consider your choices while trying to be consistent throughout the experiment.
IMPORTANT
- Some of the images are presented multiple times. Sometimes they are presented at different resolutions, whereas others are repeated. It is only natural that the quality level assigned for identical images (content and resolution) should be similar.
- The same image (identical content) present at different resolutions will often have a different quality. This is due to distortion visibility changing with the viewing scale.
- If any of the sessions are not sufficiently well-annotated, you will be asked to redo them. This is based on your accuracy and the consistency of your answers.
Let me know in YOUR BID what screen type you will use to perform the annotation: the native resolution, physical size, brand/model, and your operating system.
OVERVIEW
You will be using our browser-based app (IQAvi) to select the quality of 13,770 images. Use the slider below each image to choose a quality rating from Bad (1) to Excellent (100). Before participating in the study, please review the task instructions: https://bit.ly/iqavi-instr.
PARTICIPANT REQUIREMENTS
- experience with photography, design, or visual arts
- normal or corrected-to-normal vision
HARDWARE REQUIREMENTS
- Use a sufficiently large physical screen (15”+) with a resolution greater than 2500x1400 pixels. Do not use a smartphone or a tablet.
- Adjust your screen brightness and contrast to see each image clearly. It is possible to miss details if your screen brightness is too low relative to the ambient illumination.
- Use a fast and stable internet connection (5+ Mbps) to have little or no delay when loading images (the average image size is 2 Mb).
WORK PLAN
1. It is essential to read the task instructions (https://bit.ly/iqavi-instr) to clearly understand what technical quality is and how to use the rating scale provided.
2. A list of experiment links will be provided, which you will open one by one, from top to bottom, in your Chrome web browser. Each link (experiment session) contains a batch of 112 images.
3. Fill in your system information and enter your freelancer.com profile URL when prompted to identify your submission.
4. Click on “start experiment.”
5. Switch your browser to Full-Screen mode (bottom-right button in the IQAvi App) to maximize the visible area for large resolution images (important!)
6. Choose the quality for each image by dragging the slider at the bottom of the page.
7. When finished with a batch of images, click the submit button to get your completion code.
8. Copy and paste each completion code into the corresponding row of the provided CSV file.
9. Complete all the sessions (including training) and send the filled CSV file to me.
SUGGESTIONS
- In the training session (https://bit.ly/iqavi-10k-training), we will present you with 160 images. Half of them show the correct answer to train your understanding of the quality scale. The other half evaluates your proficiency in judging the quality of images. Use this time to practice. To continue with the main experiment, you will need to pass the training with an accuracy greater than 50% (ratio of correct to all answers). Taking the training session is not mandatory before taking up the project. Still, it is compulsory to continue with the complete experiment.
- Providing good quality annotations requires paying attention to details. Please spend sufficient time per image. It is sometimes necessary to spend up to 10 seconds to explore/pan an image to find defects that affect its quality; however, the average time per image is around 3 seconds.
- Test images are used to assess your level of attention to the task and the accuracy of your answers. Please carefully consider your choices while trying to be consistent throughout the experiment.
IMPORTANT
- Some of the images are presented multiple times. Sometimes they are presented at different resolutions, whereas others are repeated. It is only natural that the quality level assigned for identical images (content and resolution) should be similar.
- The same image (identical content) present at different resolutions will often have a different quality. This is due to distortion visibility changing with the viewing scale.
- If any of the sessions are not sufficiently well-annotated, you will be asked to redo them. This is based on your accuracy and the consistency of your answers.
Let me know in YOUR BID what screen type you will use to perform the annotation: the native resolution, physical size, brand/model, and your operating system.