Arabic LLM Prompt Injection Testing
Budget: $30 – $250 USD
I’m evaluating security vulnerabilities in three Arabic-capable language models—Allam, Falcon, and Fanar—by running the Garak prompt-injection suite. My top priority is the technical implementation, with a particular emphasis on translating each of Garak’s 256 English attack prompts into clear, natural Arabic before the tests run.
Here’s how the workflow looks:
• Build a Python notebook that loads the three models from Hugging Face (PyTorch backend), pipes the Arabic prompts through Garak, captures logits and full responses, and writes everything to tidy CSV files.
• Include bilingual testing so the notebook can toggle between the original English prompts and their Arabic counterparts, allowing side-by-side success-rate comparison.
• Produce a concise pandas analysis section that calculates and visualises attack success percentages per model and per language.
• Document every step—from model acquisition commands to translation strategy and evaluation metrics—in markdown cells so the methodology is fully reproducible.
Acceptance criteria
• Notebook runs end-to-end on a fresh environment (tested with Python 3.10, Transformers latest, and Garak).
• CSV result files and an HTML/PDF export of the notebook are generated automatically.
• Translation quality preserves the adversarial intent of each prompt; no machine-literal phrasing that could weaken the attack.
• Final section summarises Arabic vs. English success rates and highlights any model-specific patterns.
If you’re comfortable with Hugging Face Transformers, pandas, and the nuances of Arabic prompting in LLM security contexts, I’d love to collaborate and get this testing pipeline running smoothly.
Here’s how the workflow looks:
• Build a Python notebook that loads the three models from Hugging Face (PyTorch backend), pipes the Arabic prompts through Garak, captures logits and full responses, and writes everything to tidy CSV files.
• Include bilingual testing so the notebook can toggle between the original English prompts and their Arabic counterparts, allowing side-by-side success-rate comparison.
• Produce a concise pandas analysis section that calculates and visualises attack success percentages per model and per language.
• Document every step—from model acquisition commands to translation strategy and evaluation metrics—in markdown cells so the methodology is fully reproducible.
Acceptance criteria
• Notebook runs end-to-end on a fresh environment (tested with Python 3.10, Transformers latest, and Garak).
• CSV result files and an HTML/PDF export of the notebook are generated automatically.
• Translation quality preserves the adversarial intent of each prompt; no machine-literal phrasing that could weaken the attack.
• Final section summarises Arabic vs. English success rates and highlights any model-specific patterns.
If you’re comfortable with Hugging Face Transformers, pandas, and the nuances of Arabic prompting in LLM security contexts, I’d love to collaborate and get this testing pipeline running smoothly.