Development of a scientific database with web-frontend incl. meta-data
Budget: €36 – €0 EUR
Hi, I am Ulrich and I am working for the DLR.
We are currently looking to develop a scientific database (DB) with a GUI.
The DB should be searchable for metadata + tags that are available in yaml-files.
Original binary data should be linked to tabular derivatives of it (i.e. cleaned data).
In detail:
Goal
Creation of a database-based user interface for the management of scientifically used (inhomogeneous) data sets including tracking of their changes and associated (homogeneous) metadata and keywords. This also includes the creation of a suitable database schema with user management.
Data model
A distinction is made between a) raw data records, b) editable data records, c) metadata, d) keywords (tags). The linked set of these entities is called a data record. Original and derived data records may be either non-editable or editable.
- Metadata exists in a structured form. A metadata record is necessarily associated with a raw data record.
- A raw data set is a set of raw data files that are all non-editable binary data sets. A raw data set is associated with a set of editable data sets.
- Editable data sets are available in readable and editable formats. Currently, only the CSV format is supported.
- Each raw data record can be linked to any number of keywords.
- A data entry is a tuple of a metadata (record), a raw record, a set of editable records, a set of tags.
- A derived data entry is a data entry without a raw data record and instead with a reference to an original data entry and a derivation comment.
- An original data entry is a non-derived data entry, that is, a simple data entry.
- A derived editable data record is an editable data record with a reference to an original editable data record and a derivation comment
- An original editable record is a non-derivative editable record, that is, a simple editable record.
User and rights management
Login for multiple registered users with roles users and admins. Usual Rights management. Uploader becomes owner.
Search
Searching applies only to metadata and tags, not to the entire data entry.
Upload of a new data entry
The uploaded entry is automatically linked to the existing data entry (including metadata).
Database
A free, high-performance DBMS is to be used, e.g. PostgreSQL.
- SQL: for user management & organization of metadata
- NoSQL: for storing raw datasets without normalization, compatible with human-readable formats CSV & JSON as well as with small binary data (< 10 MB) in the form of dataframes from Pandas or R
- Storage of large binary datasets (>10 MB) should be done outside the database in the file system
- Upload from CSV (take first row as column name)
Technical requirements
- PostgreSQL database or comparable
- Django as web framework
Product features
Usual interfaces like login, search, upload, display search results etc. (mock-up screenshots already exist).
Product data
The size of the database can only be estimated. We expect up to 1000 records each containing about 10 to 20 tables or dataframes. An upper limit is 100,000 individual data records. Each of these data sets can range in size from a few kilobytes to 5 GB.
Each metadata sheet is in the form of YAML and has about 20 categories. Most are very limited in text length, but there is also a comment field. Up to 20 keywords can be assigned per metadata description. This specification is provided by DLR.
Technical product environment
The software must function in an environment characterized by a high level of security and a high restriction of usable software.
Other (general requirements)
An introduction to the code base is to be conducted for selected DLR employees by means of three 3-hour courses.
We are currently looking to develop a scientific database (DB) with a GUI.
The DB should be searchable for metadata + tags that are available in yaml-files.
Original binary data should be linked to tabular derivatives of it (i.e. cleaned data).
In detail:
Goal
Creation of a database-based user interface for the management of scientifically used (inhomogeneous) data sets including tracking of their changes and associated (homogeneous) metadata and keywords. This also includes the creation of a suitable database schema with user management.
Data model
A distinction is made between a) raw data records, b) editable data records, c) metadata, d) keywords (tags). The linked set of these entities is called a data record. Original and derived data records may be either non-editable or editable.
- Metadata exists in a structured form. A metadata record is necessarily associated with a raw data record.
- A raw data set is a set of raw data files that are all non-editable binary data sets. A raw data set is associated with a set of editable data sets.
- Editable data sets are available in readable and editable formats. Currently, only the CSV format is supported.
- Each raw data record can be linked to any number of keywords.
- A data entry is a tuple of a metadata (record), a raw record, a set of editable records, a set of tags.
- A derived data entry is a data entry without a raw data record and instead with a reference to an original data entry and a derivation comment.
- An original data entry is a non-derived data entry, that is, a simple data entry.
- A derived editable data record is an editable data record with a reference to an original editable data record and a derivation comment
- An original editable record is a non-derivative editable record, that is, a simple editable record.
User and rights management
Login for multiple registered users with roles users and admins. Usual Rights management. Uploader becomes owner.
Search
Searching applies only to metadata and tags, not to the entire data entry.
Upload of a new data entry
The uploaded entry is automatically linked to the existing data entry (including metadata).
Database
A free, high-performance DBMS is to be used, e.g. PostgreSQL.
- SQL: for user management & organization of metadata
- NoSQL: for storing raw datasets without normalization, compatible with human-readable formats CSV & JSON as well as with small binary data (< 10 MB) in the form of dataframes from Pandas or R
- Storage of large binary datasets (>10 MB) should be done outside the database in the file system
- Upload from CSV (take first row as column name)
Technical requirements
- PostgreSQL database or comparable
- Django as web framework
Product features
Usual interfaces like login, search, upload, display search results etc. (mock-up screenshots already exist).
Product data
The size of the database can only be estimated. We expect up to 1000 records each containing about 10 to 20 tables or dataframes. An upper limit is 100,000 individual data records. Each of these data sets can range in size from a few kilobytes to 5 GB.
Each metadata sheet is in the form of YAML and has about 20 categories. Most are very limited in text length, but there is also a comment field. Up to 20 keywords can be assigned per metadata description. This specification is provided by DLR.
Technical product environment
The software must function in an environment characterized by a high level of security and a high restriction of usable software.
Other (general requirements)
An introduction to the code base is to be conducted for selected DLR employees by means of three 3-hour courses.