Loading run…
Loading run…
ministral-14b-2512 → gemini-3.5-flash-lite · prompt v2 · retrieval embedding (gemini embeddings), top 8 · temperature 0 · 2026-10-27 11:20
Overall score
95.8
Correctness
4.82 / 5
Faithfulness
4.92 / 5
Invented answers
0%
Missed answers
5%
Expected sources cited
93%
| Category | direct | paraphrase | multi doc | partial | negative | unanswerable | trap | conflict | french |
|---|---|---|---|---|---|---|---|---|---|
| Score | 99n=15 | 94n=6 | 80n=5 | 100n=4 | 100n=5 | 100n=6 | 100n=5 | 100n=1 | 83n=3 |
Drafted answer
Yes, customer data stored in databases, file storage, and backups is encrypted at rest using AES-256. Keys are managed in AWS Key Management Service (KMS) and rotated automatically every year. Enterprise customers can provide and control their own encryption keys through AWS KMS, with revocation rendering their data unreadable by Vantrel.
Reference answer
Yes. All customer data (databases, file storage, backups) is encrypted at rest with AES-256, keys managed in AWS KMS and rotated automatically every year.
Expected: Security whitepaper › Encryption
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant correctly states that customer data is encrypted at rest using AES-256 with AWS KMS keys rotated annually, matching the reference answer. It also correctly includes the enterprise BYOK feature found in the documentation excerpts, ensuring complete coverage and high faithfulness.
Drafted answer
The minimum TLS version supported for data in transit is TLS 1.2. TLS 1.3 is preferred and negotiated when the client supports it, while older protocols like SSL, TLS 1.0, and TLS 1.1 are disabled.
Reference answer
TLS 1.2 is the minimum; TLS 1.3 is preferred. SSL, TLS 1.0 and 1.1 are disabled, and HSTS is enforced.
Expected: Security whitepaper › Encryption
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant's answer perfectly matches the reference answer regarding the minimum TLS version, preferred version, disabled protocols, and HSTS enforcement. Every single statement is fully supported by the provided documentation excerpt, and all key points of the reference are covered.
Drafted answer
Vantrel has been ISO/IEC 27001:2022 certified since September 2023. The certification covers the development, operation, and support of Vantrel Plan and is subject to annual surveillance audits.
Reference answer
Yes, ISO/IEC 27001:2022 certified since September 2023, covering development, operation and support of Vantrel Plan, with annual surveillance audits.
Expected: Security whitepaper › Certifications and audit reports
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant accurately states that the company has been ISO/IEC 27001:2022 certified since September 2023, covering the development, operation, and support of Vantrel Plan, and mentions the annual surveillance audits. All these facts perfectly align with the reference answer and are fully supported by the cited excerpt from the security whitepaper.
Drafted answer
Vantrel can share the SOC 2 Type II report with customers and prospects under a non-disclosure agreement (NDA) upon request to security@vantrel.example. The report covers the Security, Availability, and Confidentiality Trust Services Criteria for the audit period from January 1 to December 31 and is typically available in March of the following year.
Reference answer
Yes. Vantrel has an annual SOC 2 Type II (Security, Availability, Confidentiality; period Jan 1 - Dec 31, report available around March). It is shared under NDA on request to security@vantrel.example.
Expected: Security whitepaper › Certifications and audit reports
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant correctly states that a SOC 2 Type II report is available under NDA upon request to the specified email, and includes all details regarding the trust criteria and audit period. Every statement in the assistant's answer is directly supported by the cited documentation excerpts. Finally, it covers all key points of the reference answer comprehensively.
Drafted answer
Customer data for European customers is hosted in the European Union, specifically in AWS eu-west-3 (Paris, France) as the primary location with disaster recovery in AWS eu-central-1 (Frankfurt, Germany). Data at rest, including backups and disaster recovery copies, remains within the selected EU region. Limited remote access by support staff outside the EU is covered by the European Commission's Standard Contractual Clauses (SCCs) and a transfer impact assessment.
Reference answer
On AWS in eu-west-3 (Paris, France), with disaster recovery in eu-central-1 (Frankfurt, Germany). Data at rest, including backups, stays in the selected region.
Expected: Infrastructure hosting › Hosting regions
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant's answer accurately captures the primary hosting location, disaster recovery region, and the data residency guarantee stated in the reference and the documentation. It also includes the correct details regarding international data transfers and support access as supported by the citations, making it fully correct, faithful, and complete.
Drafted answer
For Vantrel Plan, our Recovery Point Objective (RPO) is 1 hour and our Recovery Time Objective (RTO) is 4 hours in the event of a major incident affecting an entire AWS region.
Reference answer
RPO of 1 hour and RTO of 4 hours.
Expected: Business continuity › Objectives
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant correctly identified the RPO as 1 hour and RTO as 4 hours, matching the reference answer precisely. The cited excerpts directly support every statement made in the answer, and all key points from the reference are included.
Drafted answer
Backups are taken continuously for databases with point-in-time recovery available for the last 7 days, and full encrypted snapshots are taken daily. These backups are retained for 35 days. Restoration of individual customer data to an earlier point in time is possible on request through support within the backup retention window.
Reference answer
Continuous backups with 7-day point-in-time recovery, plus daily encrypted full snapshots retained for 35 days, replicated to the DR location. Restores are tested monthly.
Expected: Business continuity › Backups
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant's answer covers the frequency and retention of backups accurately and matches the reference answer well. All statements are fully supported by the cited documentation excerpts, and all key points including point-in-time recovery and snapshot retention are present.
Drafted answer
Vantrel supports Single Sign-On (SSO) using SAML 2.0 and OpenID Connect (OIDC) on Business and Enterprise plans only. These protocols are tested with Microsoft Entra ID, Okta, and Google Workspace as identity providers.
Reference answer
SAML 2.0 and OpenID Connect, on Business and Enterprise plans. Tested with Microsoft Entra ID, Okta and Google Workspace. Enforcing SSO disables password login.
Expected: Identity access › Customer authentication
Grader · correctness 5 · faithfulness 5 · completeness 3
The assistant correctly identifies SAML 2.0 and OpenID Connect on Business and Enterprise plans and lists the tested identity providers, matching the reference except for omitting the detail about disabling password login, which makes it slightly incomplete.
Drafted answer
Vantrel supports SCIM 2.0 for automated user provisioning only on the Enterprise plan. Other plans require manual user management by workspace administrators.
Reference answer
Yes, SCIM 2.0 is supported, on the Enterprise plan only.
Expected: Identity access › User provisioning
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant accurately matches the reference answer by stating that SCIM 2.0 is supported for automated user provisioning exclusively on the Enterprise plan. Every statement made by the assistant is directly backed up by the provided documentation excerpt, covering all key points completely.
Drafted answer
Vantrel commits to a monthly uptime of 99.95% for Enterprise customers.
Reference answer
99.95% monthly uptime for Enterprise (excluding scheduled maintenance), with service credits of 10% or 25% of the monthly fee.
Expected: Sla support › Availability commitment
Grader · correctness 5 · faithfulness 5 · completeness 3
The assistant correctly identifies the 99.95% monthly uptime commitment for Enterprise customers, matching the primary fact in the reference answer. The answer is fully supported by the cited excerpts, although it omits the mention of service credits, making it slightly incomplete regarding all details of the reference.
Drafted answer
Vantrel notifies the customer of a personal data breach without undue delay, and in any case within 48 hours after becoming aware of it.
Reference answer
Without undue delay and in any case within 48 hours after becoming aware of it.
Expected: Data processing agreement › Personal data breach notification
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant correctly states the notification timeframe as 'without undue delay and in any case within 48 hours', perfectly matching the reference answer. Every statement in the answer is directly supported by the cited excerpt from the Data Processing Agreement, ensuring high faithfulness. Additionally, the answer is fully complete as it captures the entire key point of the reference without omitting any details.
Drafted answer
Vantrel performs penetration testing at least once a year, conducted by an independent third-party security firm, and additionally before major architectural changes. The most recent penetration test was completed in March 2026 and identified no critical findings, with two high-severity findings that were remediated within 30 days and verified by the testing firm during a retest.
Reference answer
At least annually by an independent third party, plus before major architectural changes. The last test (March 2026) found no critical and two high findings, both remediated within 30 days and verified by retest.
Expected: Vulnerability management › Penetration testing
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant's answer perfectly matches the reference answer in every detail regarding frequency, third-party status, and the results of the March 2026 test. Every statement made by the assistant is directly supported by the cited documentation excerpts. All key points from the reference are fully covered.
Drafted answer
Vantrel remediates critical vulnerabilities within 7 days. Exceptions require documented compensating controls and approval by the CISO.
Reference answer
Critical vulnerabilities must be remediated within 7 days (high: 30 days, medium: 90 days).
Expected: Vulnerability management › Vulnerability management
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant correctly identifies the 7-day remediation timeline for critical vulnerabilities, which matches the reference answer. Every statement provided is fully supported by the cited documentation excerpt, and all key points including the exception process are covered completely.
Drafted answer
Vantrel’s sub-processors are listed in the Sub-processors documents (v2026.2). For infrastructure, these include Amazon Web Services EMEA SARL (cloud hosting, storage, backups, with data location selected by the customer in EU or US), Cloudflare, Inc. (CDN, Web Application Firewall, DDoS protection, global edge network for traffic in transit), and Datadog, Inc. (infrastructure and application monitoring, logs, with data stored in EU for EU tenants and US for US tenants). For services, these include Stripe Payments Europe, Ltd. (billing and subscription payments, EU/US), Zendesk, Inc. (customer support ticketing, EU), SendGrid (Twilio Inc.) (transactional emails, US), and Microsoft Ireland Operations Ltd. (Azure OpenAI Service, AI features for Vantrel Assist, EU Data Zone). SendGrid only processes user email addresses and transactional notification content, not customer model data, and Microsoft Azure OpenAI Service is only used when Vantrel Assist is enabled by the customer.
Reference answer
AWS (hosting), Cloudflare (CDN/WAF/DDoS), Datadog (monitoring), Stripe (billing), Zendesk (support), SendGrid/Twilio (transactional emails), Microsoft Azure OpenAI Service (AI features).
Expected: Subprocessors › Infrastructure sub-processors · Subprocessors › Service sub-processors
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant provides a detailed list of sub-processors that matches the reference answer while including additional helpful context such as data locations and specific notes from the documentation. All facts are correct, completely supported by the cited excerpts, and cover all key points.
Drafted answer
No, customer data is never used to train or fine-tune AI models by Vantrel or its AI sub-processor. Only the data required to answer a specific request is sent to the model, and prompts and responses are not stored beyond the user's session, except for an audit log that does not include the content of the request.
Reference answer
No. Customer data is never used to train or fine-tune AI models, neither by Vantrel nor by its AI sub-processor.
Expected: Ai features policy › Use of customer data
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant's answer perfectly matches the reference answer in stating that customer data is never used to train AI models. Every statement made by the assistant is fully supported by the cited excerpts, and it covers all key points including the handling of prompts and responses.
Drafted answer
Yes, your employees can sign in with their corporate Okta accounts if you are on the Business or Enterprise plan, as Okta is a tested identity provider for SAML 2.0 and OpenID Connect (OIDC) Single Sign-On (SSO).
Reference answer
Yes, via SAML 2.0 or OIDC SSO on Business and Enterprise plans; Okta is a tested identity provider.
Expected: Identity access › Customer authentication
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant's answer correctly matches the reference by confirming that employees can use Okta accounts via SAML 2.0 or OIDC on Business and Enterprise plans. Every statement is directly supported by the provided documentation excerpt, and all key points from the reference are included.
Drafted answer
If the primary data center in Paris (AWS eu-west-3) were destroyed, you could lose up to 1 hour of data due to the Recovery Point Objective (RPO) of 1 hour. This is because backups are continuously replicated to the disaster recovery location in Frankfurt (AWS eu-central-1), ensuring data is synchronized within that timeframe. The loss of a single availability zone does not cause data loss, as production services run across three availability zones within the primary region.
Reference answer
At most about 1 hour of data (RPO 1 hour), with service restored from the Frankfurt DR region within 4 hours (RTO).
Expected: Business continuity › Objectives · Infrastructure hosting › Hosting regions
Grader · correctness 3 · faithfulness 5 · completeness 3
The assistant correctly identified the RPO of 1 hour matching the reference answer. However, it omitted the RTO of 4 hours mentioned in the reference answer. All statements made are fully supported by the cited excerpts.
Drafted answer
No, Vantrel engineers cannot see customer budget figures unless explicitly granted temporary access through the administration console. Access is time-limited, logged, and visible to the customer in the audit trail.
Reference answer
No standing access. Support can only access a workspace when the customer grants temporary access (max 72 hours), which is logged and visible in the audit trail.
Expected: Identity access › Vantrel employee access to customer data
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant correctly states that engineers do not have standing access and can only view customer data if temporary access is granted. The answer aligns well with the reference answer, is fully supported by the cited excerpts, and covers all key points including the time limit and audit trail.
Drafted answer
Yes, Enterprise customers can provide and control their own encryption keys for their data through AWS Key Management Service (KMS). Revoking the key makes the customer's data unreadable by Vantrel.
Reference answer
Yes, Enterprise customers can use customer-managed keys (BYOK) via AWS KMS; revoking the key makes the data unreadable by Vantrel.
Expected: Security whitepaper › Encryption
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant correctly states that Enterprise customers can use customer-managed keys via AWS KMS and that revoking them makes data unreadable. All statements are fully supported by the provided documentation excerpt, and all key points from the reference are included.
Drafted answer
Yes, Vantrel performs background checks on all new employees before their start date, where permitted by local law. All employees also sign a confidentiality agreement as part of their employment contract.
Reference answer
Yes, background checks are performed on all new employees before their start date, where permitted by local law.
Expected: Security whitepaper › Employee security
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant correctly states that background checks are performed on all new employees before their start date, matching the reference answer. The added information about confidentiality agreements is also fully supported by the cited excerpts. All criteria are met fully.
Drafted answer
For Enterprise customers, a first response to a P1 incident (service unavailable) is targeted within 1 hour, 24/7. For Business customers, the first response target is 4 hours during business hours (Monday to Friday). Starter customers receive best-effort support with no guaranteed response time.
Reference answer
Enterprise: P1 first response within 1 hour, 24/7. Business: support is business hours only, P1 first response within 4 hours.
Expected: Sla support › Support
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant correctly identifies the response times for both Enterprise and Business tiers, matching the reference answer. All stated facts are directly supported by the provided documentation excerpt. The answer is also complete, covering all key points from the reference.
Drafted answer
Yes, some of our sub-processors process data outside the EU. Stripe Payments Europe, Ltd. processes billing and subscription payment data in the US for customers hosted in the EU. SendGrid (Twilio Inc.) processes transactional emails (e.g., notifications, password resets) in the US for all customers. Datadog, Inc. processes infrastructure and application logs in the US for US-hosted tenants. Microsoft Ireland Operations Ltd. (Azure OpenAI Service) processes data only when Vantrel Assist is enabled, and it is limited to the EU Data Zone for EU-hosted customers.
Reference answer
For EU tenants, hosting (AWS), monitoring (Datadog), support (Zendesk) and AI (Azure EU Data Zone) stay in the EU. Cloudflare handles traffic in transit on its global network, Stripe operates in EU/US, and SendGrid processes email addresses and notification content in the US. Transfers are covered by SCCs; Vantrel Inc. is under the EU-U.S. Data Privacy Framework.
Expected: Subprocessors › Infrastructure sub-processors · Subprocessors › Service sub-processors · Data processing agreement › International data transfers
Grader · correctness 4 · faithfulness 5 · completeness 4
The assistant correctly identifies that some sub-processors process data outside the EU, mentioning Stripe and SendGrid accurately based on the excerpts. However, it misses AWS (which can be in the US if selected) and Cloudflare (which operates globally), while mentioning Datadog and Azure correctly with their proper regional restrictions. The statements are fully supported by the cited excerpts, and it covers most key points except for the specific detail about AWS hosting options.
Drafted answer
Vantrel informs customers of security incidents through the public status page (status.vantrel.example) for service availability incidents, with updates at least every 60 minutes for SEV1 incidents. For security incidents involving customer data, notifications are sent by email by the Security team, adhering to the timelines defined in the Data Processing Agreement. Specifically, for personal data breaches, customers are notified without undue delay and within 48 hours after Vantrel becomes aware of the breach. A post-incident report, including a root cause analysis, is shared with affected customers within 10 business days after resolution of a SEV1 incident.
Reference answer
Availability incidents on the status page (updates at least every 60 min for SEV1); customers whose data is affected are emailed by Security within the DPA timeline (48 hours); a root cause analysis is shared within 10 business days after a SEV1.
Expected: Incident response › Customer communication · Data processing agreement › Personal data breach notification
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant's answer covers all key points required by the reference answer, including status page updates, email notifications within the DPA timeline (specifying the 48-hour timeframe), and root cause analysis sharing within 10 business days. All stated facts are completely consistent with the reference and fully supported by the cited documentation excerpts.
Drafted answer
For customer users, Vantrel supports Multi-Factor Authentication (MFA) using TOTP authenticator apps or FIDO2/WebAuthn security keys. SMS-based MFA is not supported. Administrators can enforce mandatory MFA for all users in their workspace. For Vantrel staff accessing production environments, access is granted only through temporary, time-limited (maximum 72 hours) customer-granted permissions, which are logged and auditable. Vantrel employees do not have standing access to customer data or production accounts, and access to internal systems is reviewed quarterly with revocation within 24 hours of an employee leaving the company.
Reference answer
Customers: TOTP or FIDO2/WebAuthn, can be made mandatory by admins, no SMS. Staff: production admin access goes through a bastion with SSO, MFA and session recording; direct SSH disabled.
Expected: Identity access › Customer authentication · Infrastructure hosting › Network security
Grader · correctness 1 · faithfulness 3 · completeness 1
The assistant accurately described the MFA options for customer users, but completely missed the MFA and bastion requirements for Vantrel staff accessing production, describing customer data access instead. The customer part is fully supported by the excerpts and correct, but the production staff access part is completely wrong regarding the reference answer.
Drafted answer
Infrastructure and application logs are stored in Datadog, with EU-hosted tenants using the EU site and US-hosted tenants using the US site. Infrastructure logs are retained for 12 months, as documented. Audit logs are retained for 13 months and can be exported via API or viewed in the administration console, with Enterprise customers able to stream them to their SIEM via webhook. The retention period for audit logs is not explicitly stated for Vantrel Plan customers outside of the 13-month duration.
Reference answer
Infrastructure and application logs are centralized in Datadog (EU site for EU tenants), retained 12 months. Customer audit logs are retained 13 months and exportable via API or streamed to a SIEM (Enterprise).
Expected: Infrastructure hosting › Logging and monitoring · Identity access › Audit logs
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant's answer closely matches the reference answer and accurately reflects the provided documentation excerpts. It covers infrastructure and application logs in Datadog, their retention, and audit log retention and export features. All statements are fully supported by the cited excerpts.
Drafted answer
Yes, when Vantrel Assist is enabled, customer data is sent to Microsoft Ireland Operations Ltd. (Azure OpenAI Service) in the EU Data Zone for EU-hosted tenants only. Only the data required to answer a specific request (e.g., selected cells and their labels) is sent, and it is never used to train or fine-tune AI models. Data is not stored beyond the user's session except in the audit log, which records AI feature usage without storing request content.
Reference answer
Only if a workspace Admin enables Vantrel Assist (off by default). Then only the data needed for the request is sent to Microsoft Azure OpenAI Service in the EU Data Zone; it is never used for training.
Expected: Ai features policy › Vantrel Assist · Ai features policy › Use of customer data · Subprocessors › Service sub-processors
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant's answer accurately captures all essential details from the reference answer, including who the third party is, where the data goes, the opt-in requirement, the lack of training on customer data, and data retention specifics. Every statement is fully supported by the provided documentation excerpts.
Drafted answer
Vantrel operates a private bug bounty program with invited researchers only. The documentation does not specify reward amounts for reported vulnerabilities.
Reference answer
Yes, a private bug bounty program with invited researchers. Reward amounts are not documented.
Expected: Incident response › Reporting a vulnerability
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant correctly identified that the company runs a private bug bounty program with invited researchers and that reward amounts are not documented, matching the reference answer completely. All statements are fully supported by the provided excerpt, and all key points are covered.
Drafted answer
The name of the firm that performed the last penetration test is not in the documentation.
Reference answer
An independent third-party security firm, in March 2026. The firm's name is not documented.
Expected: Vulnerability management › Penetration testing
Grader · correctness 5 · faithfulness 5 · completeness 5
The reference answer states that the firm's name is not documented, and the assistant correctly identified that the information is not in the documentation. Therefore, all criteria receive the maximum score.
Drafted answer
Vantrel supports streaming audit logs to a customer's SIEM via webhook for Enterprise customers only. The documentation does not specify compatibility with particular SIEM products, only that the integration is possible via webhook.
Reference answer
Enterprise customers can stream audit logs to their SIEM via webhook; specific supported SIEM products are not listed.
Expected: Identity access › Audit logs
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant accurately captures that Enterprise customers can stream audit logs to their SIEM via webhook and correctly notes that specific SIEM products are not listed, matching the reference answer. All statements are fully supported by the provided documentation excerpt, and all key points are completely covered.
Drafted answer
The Data Protection Officer (DPO) at Vantrel can be contacted at dpo@vantrel.example. The DPO is part of the Security Committee led by the Chief Information Security Officer (CISO).
Reference answer
The DPO can be contacted at dpo@vantrel.example; the DPO's name is not documented.
Expected: Company overview › Security and compliance organization
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant correctly provided the DPO email address found in the excerpts, while appropriately noting that the DPO's name is not mentioned (aligning with the reference answer). All stated facts are directly supported by the cited excerpts, and the key points from the reference are fully covered.
Drafted answer
No, Vantrel Plan does not currently support custom roles. Only the four default roles (Admin, Modeler, Contributor, Viewer) are available, and access can be restricted at the dimension-member level.
Reference answer
No, custom roles are not currently supported. Four default roles exist (Admin, Modeler, Contributor, Viewer) plus dimension-member-level restrictions.
Expected: Identity access › Roles and permissions
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant's answer perfectly matches the reference answer by stating that custom roles are not supported and listing the four default roles along with the dimension-member level restrictions. Every statement is directly supported by the cited excerpts, and all key points from the reference are fully covered.
Drafted answer
No, Vantrel does not offer on-premise, private cloud, or self-hosted deployments. Vantrel Plan is delivered exclusively as a multi-tenant Software-as-a-Service (SaaS) application.
Reference answer
No. Vantrel Plan is SaaS only; no on-premise, private cloud or self-hosted deployment.
Expected: Company overview › Product and delivery model
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant correctly states that Vantrel does not offer on-premise deployments, matching the reference answer. Every statement is directly supported by the cited documentation excerpt, and all key points from the reference are fully covered.
Drafted answer
No, Vantrel does not support SMS-based multi-factor authentication for customer users.
Reference answer
No, SMS-based MFA is not supported (TOTP and FIDO2/WebAuthn are).
Expected: Identity access › Customer authentication
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant correctly states that SMS-based MFA is not supported, perfectly matching the reference answer. Every statement in the answer is directly supported by the cited documentation excerpt. Finally, it covers the key point completely without missing any important details.
Drafted answer
The documentation does not mention hosting in Australia.
Reference answer
No. Only EU (Paris/Frankfurt) and US (N. Virginia/Oregon) regions are offered.
Expected: Infrastructure hosting › Hosting regions
Grader · correctness 5 · faithfulness 5 · completeness 5
The reference answer states that data cannot be hosted in Australia, as only EU and US regions are offered. The assistant correctly identified that the documentation does not mention hosting in Australia. Therefore, the assistant receives top scores across all criteria.
Drafted answer
No, Vantrel does not currently have an accessibility conformance report (VPAT). We are working toward conformance with WCAG 2.1 level AA.
Reference answer
Not yet. Vantrel is working toward WCAG 2.1 AA, but a VPAT is not available yet.
Expected: Product faq › Is Vantrel Plan accessible to users with disabilities?
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant correctly states that a VPAT is not currently available and mentions the work towards WCAG 2.1 level AA. This matches the reference answer and is fully supported by the cited documentation excerpt.
Drafted answer
The documentation does not mention whether Vantrel Plan is FedRAMP authorized.
Reference answer
Not in the documentation.
Grader · correctness 5 · faithfulness 5 · completeness 5
The reference answer states that the information is not in the documentation, and the assistant correctly identified that the documentation does not mention this, giving an accurate response without hallucinating.
Drafted answer
The documentation does not address whether Vantrel will sign a HIPAA Business Associate Agreement (BAA).
Reference answer
Not in the documentation.
Grader · correctness 5 · faithfulness 5 · completeness 5
The reference answer states that the information is not in the documentation. The assistant correctly identifies that the documentation does not address the question, matching the reference and scoring 5 across all criteria.
Drafted answer
The coverage amount of the cyber insurance policy is not in the documentation.
Reference answer
Not in the documentation.
Grader · correctness 5 · faithfulness 5 · completeness 5
The reference answer states that the information is not in the documentation, and the assistant correctly answers that the coverage amount is not found in the documentation. Since no documentation is available and the assistant correctly identifies this, all scores are the maximum value.
Drafted answer
The documentation does not mention whether Vantrel offers a source code escrow agreement.
Reference answer
Not in the documentation.
Grader · correctness 5 · faithfulness 5 · completeness 5
The reference answer states that the information is not in the documentation. The assistant correctly identified that the documentation does not mention source code escrow agreements, meaning all scores are at the maximum level.
Drafted answer
The documentation does not mention IP allowlisting to restrict user access.
Reference answer
Not in the documentation.
Grader · correctness 5 · faithfulness 5 · completeness 5
The reference answer states that the information is not in the documentation, and the assistant correctly identified this and stated that the documentation does not mention IP allowlisting. Therefore, all criteria achieve the maximum score.
Drafted answer
The documentation does not mention whether Vantrel uses post-quantum cryptography.
Reference answer
Not in the documentation.
Grader · correctness 5 · faithfulness 5 · completeness 5
The reference answer states that the information is not in the documentation. The assistant correctly identifies that the documentation does not mention post-quantum cryptography, perfectly matching the reference.
Drafted answer
The documentation does not mention whether a SOC 1 report is available.
Reference answer
Not in the documentation. Only Vantrel's SOC 2 Type II is documented; the SOC 1 mentioned is AWS's, not Vantrel's.
Grader · correctness 5 · faithfulness 5 · completeness 5
The reference answer states that the information is not documented, and the assistant correctly identified that the documentation does not mention a SOC 1 report. Therefore, all criteria receive the maximum score.
Drafted answer
The documentation does not specify the number of people working in the security team.
Reference answer
Not in the documentation. Only the total headcount (~240) and the CISO/Security Committee are mentioned.
Grader · correctness 5 · faithfulness 5 · completeness 5
The reference answer states that the information is not in the documentation, and the assistant correctly identifies that the documentation does not specify the number of people working in the security team. Therefore, all criteria are fully met.
Drafted answer
The documentation does not specify the mean time to resolve P1 incidents.
Reference answer
Not in the documentation. Only first-response targets are documented, explicitly not resolution times.
Grader · correctness 5 · faithfulness 5 · completeness 5
The reference answer states that the information is not in the documentation, and the assistant correctly identified this and stated that the documentation does not specify the mean time to resolve P1 incidents. Therefore, all criteria score the maximum value of 5.
Drafted answer
The documentation does not mention CSA STAR certification.
Reference answer
Not in the documentation. The documented certifications are ISO 27001 and SOC 2 Type II.
Grader · correctness 5 · faithfulness 5 · completeness 5
The reference answer states that the information is not in the documentation, and the assistant correctly identifies that the documentation does not mention CSA STAR certification. Therefore, all criteria receive the maximum score of 5.
Drafted answer
Vantrel does not provide an emergency phone number for security incidents in the documentation.
Reference answer
Not in the documentation. Only email (security@vantrel.example) and the status page are documented.
Grader · correctness 5 · faithfulness 5 · completeness 5
The reference answer states that the information is not documented and mentions the alternative email contact. The assistant correctly identifies that no emergency phone number is provided in the documentation, matching the expected outcome for a missing information scenario.
Drafted answer
After contract termination, customers have 30 days to export their data. According to the Security Whitepaper, customer data is permanently deleted from production systems within 90 days of contract termination, while the Data Processing Agreement states it is deleted within 30 days after the end of the 30-day grace period. These two sources conflict on the deletion timeline.
Reference answer
The documentation is inconsistent: the Security Whitepaper says within 90 days of termination, while the DPA says within 30 days after a 30-day export grace period (i.e. ~60 days). Both sources should be cited and the inconsistency flagged.
Expected: Security whitepaper › Data retention and deletion · Data processing agreement › Return and deletion of data
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant correctly identified the inconsistency between the Security Whitepaper (90 days) and the DPA (30 days after a 30-day grace period) using the provided documentation, matching the reference answer completely. Every statement made by the assistant is fully supported by the cited excerpts, and all key points regarding the timeline and the conflict are covered.
Drafted answer
Les données des clients hébergés dans la région EU sont stockées dans la région AWS eu-west-3 (Paris, France) pour le site principal, mais des copies de sauvegarde et de reprise après sinistre sont également stockées dans AWS eu-central-1 (Francfort, Allemagne). Aucune information dans les extraits ne précise si les données sont strictement limitées à la France ou si elles peuvent être répliquées ou traitées en dehors de la France dans le cadre des opérations de sauvegarde ou de reprise après sinistre.
Reference answer
Oui pour la région EU (par défaut) : AWS eu-west-3 à Paris, avec reprise d'activité à Francfort (Allemagne). Une région US existe aussi.
Expected: Infrastructure hosting › Hosting regions
Grader · correctness 3 · faithfulness 3 · completeness 3
The assistant correctly identified that data in the EU region is hosted in Paris, France, but incorrectly stated that there is no information in the excerpts regarding backups being limited to the region, whereas the excerpt explicitly states that customer data at rest stays within the selected region including backups. This leads to a lower faithfulness and completeness score due to missing the key point about data staying within the region.
Drafted answer
Oui, Vantrel Plan propose l’authentification unique (SSO) avec Microsoft Entra ID via SAML 2.0 et OpenID Connect (OIDC), mais uniquement sur les plans Business et Enterprise.
Reference answer
Oui, SSO SAML 2.0 et OIDC sur les plans Business et Enterprise ; Microsoft Entra ID fait partie des fournisseurs testés.
Expected: Identity access › Customer authentication
Grader · correctness 5 · faithfulness 5 · completeness 5
The assistant's answer correctly matches the reference answer by confirming that SSO with Microsoft Entra ID is available via SAML 2.0 and OIDC on Business and Enterprise plans. Every statement made by the assistant is fully supported by the provided documentation excerpts, and all key points from the reference are included.
Drafted answer
L'information concernant une qualification SecNumCloud par l'ANSSI n'est pas présente dans les extraits de documentation fournis.
Reference answer
Pas dans la documentation.
Grader · correctness 5 · faithfulness 5 · completeness 5
The reference answer states that the information is not in the documentation. The assistant correctly identifies that the information is absent from the provided excerpts, aligning with the reference.