Drafted by Gemini, graded by Gemini

Drafted answer

Yes, all customer data stored in databases, file storage, and backups is encrypted at rest using AES-256. Keys are managed in AWS Key Management Service (KMS) and are rotated automatically every year. Additionally, Enterprise customers have the option to provide and control their own encryption key through AWS KMS (BYOK).

• ↳ Security Whitepaper › Encryption

Reference answer

Yes. All customer data (databases, file storage, backups) is encrypted at rest with AES-256, keys managed in AWS KMS and rotated automatically every year.

Expected: Security whitepaper › Encryption

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant accurately answers the question covering all key points (AES-256, AWS KMS, annual rotation) and even includes the optional BYOK feature mentioned in the excerpts. All statements are fully supported by the cited excerpts, matching the reference answer well.

Drafted answer

All connections to Vantrel Plan use TLS 1.2 or higher, with TLS 1.3 preferred and negotiated whenever supported by the client. Older protocols such as SSL, TLS 1.0, and 1.1 are disabled.

• ↳ Security Whitepaper › Encryption

Reference answer

TLS 1.2 is the minimum; TLS 1.3 is preferred. SSL, TLS 1.0 and 1.1 are disabled, and HSTS is enforced.

Expected: Security whitepaper › Encryption

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant correctly states that TLS 1.2 is the minimum and TLS 1.3 is preferred, matching the reference answer. All statements are fully supported by the cited excerpt. However, it misses the detail regarding the enforcement of HSTS present in the reference answer.

Drafted answer

Yes, Vantrel has been certified against ISO/IEC 27001:2022 since September 2023. The certificate covers the development, operation, and support of Vantrel Plan.

• ↳ Security Whitepaper › Certifications and audit reports

Reference answer

Yes, ISO/IEC 27001:2022 certified since September 2023, covering development, operation and support of Vantrel Plan, with annual surveillance audits.

Expected: Security whitepaper › Certifications and audit reports

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant correctly states that Vantrel has been ISO/IEC 27001:2022 certified since September 2023 and covers the development, operation, and support of Vantrel Plan, which matches the reference answer and is fully supported by the cited excerpts, though it omits the mention of annual surveillance audits.

Drafted answer

Yes, Vantrel undergoes an annual SOC 2 Type II examination covering Security, Availability, and Confidentiality Trust Services Criteria, and the report can be shared with customers and prospects under a non-disclosure agreement (NDA) upon request to security@vantrel.example.

• ↳ Security Whitepaper › Certifications and audit reports

Reference answer

Yes. Vantrel has an annual SOC 2 Type II (Security, Availability, Confidentiality; period Jan 1 - Dec 31, report available around March). It is shared under NDA on request to security@vantrel.example.

Expected: Security whitepaper › Certifications and audit reports

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant accurately provides all necessary information about the SOC 2 Type II report, matching the reference answer completely. Every statement made by the assistant is directly supported by the cited excerpts from the documentation, and all key points regarding trust criteria, frequency, availability, and NDA requests are covered.

Drafted answer

For European customers, customer data is hosted in the EU region (AWS eu-west-3 in Paris, France), with disaster recovery copies in AWS eu-central-1 (Frankfurt, Germany).

• ↳ Infrastructure and Hosting › Hosting regions

Reference answer

On AWS in eu-west-3 (Paris, France), with disaster recovery in eu-central-1 (Frankfurt, Germany). Data at rest, including backups, stays in the selected region.

Expected: Infrastructure hosting › Hosting regions

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant accurately states the hosting region and disaster recovery location for European customers, matching both the reference answer and the cited documentation. All facts are fully supported by the excerpts and cover the necessary key points.

Drafted answer

For Vantrel Plan, the Recovery Point Objective (RPO) is 1 hour and the Recovery Time Objective (RTO) is 4 hours.

• ↳ Business Continuity and Disaster Recovery › Objectives

Reference answer

RPO of 1 hour and RTO of 4 hours.

Expected: Business continuity › Objectives

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identified the RPO as 1 hour and RTO as 4 hours, matching the reference answer. All statements are fully supported by the cited excerpts from the documentation, and the answer is complete.

Drafted answer

Databases are backed up continuously with point-in-time recovery available for the last 7 days, and full encrypted snapshots are taken daily and retained for 35 days.

• ↳ Business Continuity and Disaster Recovery › Backups

Reference answer

Continuous backups with 7-day point-in-time recovery, plus daily encrypted full snapshots retained for 35 days, replicated to the DR location. Restores are tested monthly.

Expected: Business continuity › Backups

Grader · correctness 5 · faithfulness 5 · completeness 4

The assistant correctly details the continuous backup frequency, 7-day point-in-time recovery, daily snapshots, and 35-day retention as stated in the reference. Every statement is fully supported by the cited excerpts, although it omits the monthly testing and replication details, resulting in a slightly lower completeness score.

Drafted answer

Vantrel supports SAML 2.0 and OpenID Connect (OIDC) Single Sign-On (SSO) protocols, which are available on Business and Enterprise plans.

• ↳ Identity and Access Management › Customer authentication

Reference answer

SAML 2.0 and OpenID Connect, on Business and Enterprise plans. Tested with Microsoft Entra ID, Okta and Google Workspace. Enforcing SSO disables password login.

Expected: Identity access › Customer authentication

Grader · correctness 4 · faithfulness 5 · completeness 4

The assistant correctly identified the supported SSO protocols and plan restrictions based on the documentation, matching the reference answer well. Every statement in the answer is directly supported by the cited excerpts. It missed the minor detail about enforcing SSO disabling password login, but captures the core information.

Drafted answer

Yes, user provisioning and deprovisioning through SCIM 2.0 is supported on the Enterprise plan.

• ↳ Identity and Access Management › User provisioning

Reference answer

Yes, SCIM 2.0 is supported, on the Enterprise plan only.

Expected: Identity access › User provisioning

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly states that SCIM 2.0 is supported for user provisioning and deprovisioning on the Enterprise plan, which fully matches the reference answer. Every statement is directly supported by the cited documentation excerpt, and all key points from the reference are included without any missing details.

Drafted answer

For Enterprise customers, we commit to a monthly uptime of 99.95%, excluding scheduled maintenance.

• ↳ Service Level Agreement and Support › Availability commitment

Reference answer

99.95% monthly uptime for Enterprise (excluding scheduled maintenance), with service credits of 10% or 25% of the monthly fee.

Expected: Sla support › Availability commitment

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant correctly identified the 99.95% monthly uptime commitment for Enterprise customers, matching the primary fact in the reference answer. All statements are fully supported by the cited documentation excerpt. However, it omitted the service credits detail which was present in the reference answer, making completeness slightly lower.

Drafted answer

Vantrel notifies the customer of any personal data breach affecting the customer's data without undue delay, and in any case within 48 hours after becoming aware of it.

• ↳ Data Processing Agreement (summary) › Personal data breach notification
• ↳ Incident Response › Customer communication (not in expected sources)

Reference answer

Without undue delay and in any case within 48 hours after becoming aware of it.

Expected: Data processing agreement › Personal data breach notification

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer perfectly matches the reference answer by stating that notification will occur without undue delay and within 48 hours. Every statement is directly supported by the cited excerpts from the Data Processing Agreement, and all key points from the reference are fully covered.

Drafted answer

Vantrel Plan is tested by an independent third-party security firm at least once a year, and additionally before major architectural changes. The most recent penetration test was completed in March 2026 and identified no critical findings and two high-severity findings, both of which were remediated within 30 days and verified by a retest.

• ↳ Penetration Testing and Vulnerability Management › Penetration testing

Reference answer

At least annually by an independent third party, plus before major architectural changes. The last test (March 2026) found no critical and two high findings, both remediated within 30 days and verified by retest.

Expected: Vulnerability management › Penetration testing

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer fully matches the reference answer in terms of facts, covering the frequency of penetration testing and the specific results of the March 2026 test. Every statement made by the assistant is directly supported by the provided excerpts, and all key points from the reference are included.

Drafted answer

Vantrel remediates critical vulnerabilities within 7 days.

• ↳ Penetration Testing and Vulnerability Management › Vulnerability management

Reference answer

Critical vulnerabilities must be remediated within 7 days (high: 30 days, medium: 90 days).

Expected: Vulnerability management › Vulnerability management

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identified the 7-day remediation timeline for critical vulnerabilities, which matches the reference answer. The statement is fully supported by the cited documentation excerpt. It covers the specific key point requested about critical vulnerabilities.

Drafted answer

Vantrel uses the following infrastructure sub-processors: Amazon Web Services EMEA SARL, Cloudflare, Inc., and Datadog, Inc. The service sub-processors are Stripe Payments Europe, Ltd., Zendesk, Inc., SendGrid (Twilio Inc.), and Microsoft Ireland Operations Ltd. (Azure OpenAI Service).

• ↳ Sub-processors › Infrastructure sub-processors
• ↳ Sub-processors › Service sub-processors

Reference answer

AWS (hosting), Cloudflare (CDN/WAF/DDoS), Datadog (monitoring), Stripe (billing), Zendesk (support), SendGrid/Twilio (transactional emails), Microsoft Azure OpenAI Service (AI features).

Expected: Subprocessors › Infrastructure sub-processors · Subprocessors › Service sub-processors

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant accurately lists all the sub-processors mentioned in the reference answer and supported by the cited documentation excerpts. All facts match the reference, everything is fully supported by the citations, and all key points are covered.

Drafted answer

Customer data is never used to train or fine-tune AI models, neither by Vantrel nor by its AI sub-processor.

• ↳ AI Features Policy › Use of customer data

Reference answer

No. Customer data is never used to train or fine-tune AI models, neither by Vantrel nor by its AI sub-processor.

Expected: Ai features policy › Use of customer data

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer directly matches the reference answer, stating that customer data is never used to train or fine-tune AI models. Every statement made by the assistant is fully supported by the cited excerpts, and all key points of the reference are covered.

Drafted answer

Yes, Single Sign-On (SSO) via OpenID Connect and SAML 2.0 is supported on Business and Enterprise plans, and Okta is one of the tested identity providers.

• ↳ Identity and Access Management › Customer authentication

Reference answer

Yes, via SAML 2.0 or OIDC SSO on Business and Enterprise plans; Okta is a tested identity provider.

Expected: Identity access › Customer authentication

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly states that employees can sign in with Okta using SAML 2.0 or OIDC on Business and Enterprise plans, matching the reference answer. Every fact is directly supported by the provided documentation excerpt, resulting in full scores across all criteria.

Drafted answer

In the event of a major incident affecting an entire AWS region such as Paris, customer data at rest stays within the selected region and is replicated to the disaster recovery location, with a defined Recovery Point Objective (RPO) of 1 hour.

• ↳ Infrastructure and Hosting › Hosting regions
• ↳ Business Continuity and Disaster Recovery › Backups (not in expected sources)
• ↳ Business Continuity and Disaster Recovery › Objectives

Reference answer

At most about 1 hour of data (RPO 1 hour), with service restored from the Frankfurt DR region within 4 hours (RTO).

Expected: Business continuity › Objectives · Infrastructure hosting › Hosting regions

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant correctly identifies the RPO of 1 hour based on the documentation excerpts and explains the regional setup, though it omits the RTO (Recovery Time Objective) of 4 hours mentioned in the reference answer. Every statement provided in the answer is fully supported by the cited excerpts.

Drafted answer

Vantrel employees do not have standing access to customer data. Support engineers can only access a customer workspace when the customer grants temporary access from the administration console.

• ↳ Identity and Access Management › Vantrel employee access to customer data

Reference answer

No standing access. Support can only access a workspace when the customer grants temporary access (max 72 hours), which is logged and visible in the audit trail.

Expected: Identity access › Vantrel employee access to customer data

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer accurately captures the main points from the reference answer, confirming that engineers do not have standing access and require temporary, customer-granted access. All statements are fully supported by the cited documentation excerpt, and the key details regarding access limits and auditing are present.

Drafted answer

Yes, Enterprise customers can provide and control their own encryption key through AWS KMS (Bring Your Own Key), which allows them to make their data unreadable by Vantrel if the key is revoked.

• ↳ Security Whitepaper › Encryption

Reference answer

Yes, Enterprise customers can use customer-managed keys (BYOK) via AWS KMS; revoking the key makes the data unreadable by Vantrel.

Expected: Security whitepaper › Encryption

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant accurately answers the question matching all facts from the reference answer. Every statement made by the assistant is directly supported by the cited documentation excerpt on customer-managed keys. The answer is complete and covers all key points including the customer tier, the technology used, and the effect of revocation.

Drafted answer

Background checks are performed on all new employees before their start date, where permitted by local law.

• ↳ Security Whitepaper › Employee security

Reference answer

Yes, background checks are performed on all new employees before their start date, where permitted by local law.

Expected: Security whitepaper › Employee security

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer perfectly matches the reference answer, is fully supported by the cited documentation excerpt, and covers all key points completely.

Drafted answer

Response times depend on your subscription plan. For Enterprise customers, the first-response target for P1 (service unavailable) is 1 hour, 24/7. For Business customers, the target is 4 hours during business hours (Monday–Fri). Starter customers have access to email support and the help center on a best-effort basis.

• ↳ Service Level Agreement and Support › Support

Reference answer

Enterprise: P1 first response within 1 hour, 24/7. Business: support is business hours only, P1 first response within 4 hours.

Expected: Sla support › Support

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer completely matches the reference answer by specifying the response times for both Enterprise and Business plans. All statements are fully supported by the cited documentation excerpts, and no key points from the reference are missed.

Drafted answer

Yes. Sub-processors such as Stripe Payments Europe, Ltd. and SendGrid (Twilio Inc.) process data in the US, and Cloudflare, Inc. operates on a global edge network. Additionally, for customers with US-hosted tenants or depending on the selected services, infrastructure and monitoring data may be processed in the US.

• ↳ Sub-processors › Service sub-processors
• ↳ Sub-processors › Infrastructure sub-processors

Reference answer

For EU tenants, hosting (AWS), monitoring (Datadog), support (Zendesk) and AI (Azure EU Data Zone) stay in the EU. Cloudflare handles traffic in transit on its global network, Stripe operates in EU/US, and SendGrid processes email addresses and notification content in the US. Transfers are covered by SCCs; Vantrel Inc. is under the EU-U.S. Data Privacy Framework.

Expected: Subprocessors › Infrastructure sub-processors · Subprocessors › Service sub-processors · Data processing agreement › International data transfers

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly identifies that some sub-processors operate outside the EU (like SendGrid and Cloudflare), which matches the reference answer and the cited excerpts. The facts are fully supported by the citations, and the key points regarding data processing locations are covered comprehensively.

Drafted answer

Customers affected by a security incident involving their data are notified by email by the Security team, adhering to the timelines defined in the Data Processing Agreement, which specifies notification of a personal data breach without undue delay and within 48 hours after becoming aware of it. Additionally, service availability incidents are published on the public status page at status.vantrel.example.

• ↳ Incident Response › Customer communication
• ↳ Data Processing Agreement (summary) › Personal data breach notification

Reference answer

Availability incidents on the status page (updates at least every 60 min for SEV1); customers whose data is affected are emailed by Security within the DPA timeline (48 hours); a root cause analysis is shared within 10 business days after a SEV1.

Expected: Incident response › Customer communication · Data processing agreement › Personal data breach notification

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer covers all key points from the reference, including status page updates, email notifications within the DPA timeline, and root cause analysis sharing, while remaining fully supported by the cited excerpts.

Drafted answer

For customer users, multi-factor authentication (MFA) can be enabled using a TOTP authenticator app or a FIDO2/WebAuthn security key, and administrators can make MFA mandatory for all users of their workspace (SMS-based MFA is not supported). The documentation does not contain information regarding MFA options for Vantrel staff accessing production.

• ↳ Identity and Access Management › Customer authentication

Reference answer

Customers: TOTP or FIDO2/WebAuthn, can be made mandatory by admins, no SMS. Staff: production admin access goes through a bastion with SSO, MFA and session recording; direct SSH disabled.

Expected: Identity access › Customer authentication · Infrastructure hosting › Network security

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant correctly answered the customer MFA options based on the excerpts, matching the reference details for customers. It also correctly noted that the documentation does not contain information regarding Vantrel staff production access, which aligns with the fact that the provided excerpts lack that information while the reference answer includes it.

Drafted answer

Infrastructure and application logs are centralized in Datadog, utilizing the EU site for EU-hosted tenants, and are retained for 12 months. Audit logs are retained for 13 months, but the documentation does not specify where audit logs are stored. Additionally, there is a conflict in the documentation regarding data deletion timelines after contract termination (30 days versus 90 days), though this specifically refers to customer data rather than logs.

• ↳ Infrastructure and Hosting › Logging and monitoring
• ↳ Identity and Access Management › Audit logs
• ↳ Security Whitepaper › Data retention and deletion (not in expected sources)
• ↳ Sub-processors › Infrastructure sub-processors (not in expected sources)
• ↳ Data Processing Agreement (summary) › Return and deletion of data (not in expected sources)

Reference answer

Infrastructure and application logs are centralized in Datadog (EU site for EU tenants), retained 12 months. Customer audit logs are retained 13 months and exportable via API or streamed to a SIEM (Enterprise).

Expected: Infrastructure hosting › Logging and monitoring · Identity access › Audit logs

Grader · correctness 4 · faithfulness 5 · completeness 4

The assistant correctly states the storage location and retention period for infrastructure/application logs and audit logs, but misses the export/SIEM details for audit logs present in the reference. Every statement is fully supported by the cited excerpts. The assistant also correctly flags the conflicting sources regarding data deletion timelines, aligning with the evaluation criteria.

Drafted answer

Yes, when Vantrel Assist is enabled, customer data required to answer a specific request is sent to Microsoft Ireland Operations Ltd. (Azure OpenAI Service), which serves as the AI sub-processor. For EU-hosted tenants, this service is deployed in the EU Data Zone.

• ↳ AI Features Policy › Use of customer data
• ↳ Sub-processors › Service sub-processors
• ↳ AI Features Policy › Vantrel Assist
• ↳ Sub-processors › Notes (not in expected sources)

Reference answer

Only if a workspace Admin enables Vantrel Assist (off by default). Then only the data needed for the request is sent to Microsoft Azure OpenAI Service in the EU Data Zone; it is never used for training.

Expected: Ai features policy › Vantrel Assist · Ai features policy › Use of customer data · Subprocessors › Service sub-processors

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant correctly identifies that data is sent to Microsoft Azure OpenAI Service in the EU Data Zone, matching the reference answer. However, it misses the crucial point that Vantrel Assist is off by default and requires an admin to enable it. All statements are supported by the provided excerpts.

Drafted answer

Vantrel runs a private bug bounty program with invited researchers as stated in the documentation. However, the documentation does not contain any information regarding reward amounts.

• ↳ Incident Response › Reporting a vulnerability

Reference answer

Yes, a private bug bounty program with invited researchers. Reward amounts are not documented.

Expected: Incident response › Reporting a vulnerability

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that there is a private bug bounty program with invited researchers and accurately states that reward amounts are not documented. All statements are fully supported by the provided documentation excerpt and cover all key points of the reference answer.

Drafted answer

The documentation states that Vantrel is tested by an independent third-party security firm at least once a year, but the specific name or identity of the firm is not documented.

• ↳ Penetration Testing and Vulnerability Management › Penetration testing

Reference answer

An independent third-party security firm, in March 2026. The firm's name is not documented.

Expected: Vulnerability management › Penetration testing

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant correctly notes that an independent third-party firm performed the test and that the specific name is not documented, matching the reference answer well. However, it omits the mention of the test date (March 2026), which affects completeness. All statements are fully supported by the cited excerpts.

Drafted answer

Enterprise customers can stream audit logs to their SIEM via webhook.

• ↳ Identity and Access Management › Audit logs

Reference answer

Enterprise customers can stream audit logs to their SIEM via webhook; specific supported SIEM products are not listed.

Expected: Identity access › Audit logs

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly states that Enterprise customers can stream audit logs to their SIEM via webhook, aligning with the reference answer and the cited documentation. Every statement in the answer is directly supported by the excerpt, and all key points are covered without including unsupported details.

Drafted answer

The Data Protection Officer (DPO) can be contacted at dpo@vantrel.example.

• ↳ Company Overview › Security and compliance organization

Reference answer

The DPO can be contacted at dpo@vantrel.example; the DPO's name is not documented.

Expected: Company overview › Security and compliance organization

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly provided the DPO's contact email as found in the documentation, and omitted the name since it is not documented, matching the reference answer. All statements are fully supported by the cited excerpts, and all key points from the reference are covered.

Drafted answer

No, custom roles are not currently supported by Vantrel Plan.

• ↳ Identity and Access Management › Roles and permissions

Reference answer

No, custom roles are not currently supported. Four default roles exist (Admin, Modeler, Contributor, Viewer) plus dimension-member-level restrictions.

Expected: Identity access › Roles and permissions

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly states that custom roles are not supported, matching the reference answer. Every statement in the answer is fully supported by the cited excerpts, and all key points from the reference are included.

Drafted answer

Vantrel does not offer on-premise, private cloud or self-hosted deployments. Vantrel Plan is delivered exclusively as a multi-tenant Software-as-a-Service (SaaS) application accessed through a web browser.

• ↳ Company Overview › Product and delivery model

Reference answer

No. Vantrel Plan is SaaS only; no on-premise, private cloud or self-hosted deployment.

Expected: Company overview › Product and delivery model

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly states that Vantrel does not offer on-premise deployments, matching both the reference answer and the cited documentation. Every statement is fully supported by the provided excerpts, and all key points regarding the SaaS-only delivery model are included.

Drafted answer

SMS-based MFA is not supported.

• ↳ Identity and Access Management › Customer authentication

Reference answer

No, SMS-based MFA is not supported (TOTP and FIDO2/WebAuthn are).

Expected: Identity access › Customer authentication

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly answered that SMS-based MFA is not supported, matching the reference answer precisely. The statement is fully supported by the provided documentation excerpt. All key points from the reference are included.

Drafted answer

No, data cannot be hosted in Australia. The available hosting regions are EU (Paris, France with disaster recovery in Frankfurt, Germany) and US (N. Virginia with disaster recovery in Oregon), and no other hosting regions are currently offered.

• ↳ Infrastructure and Hosting › Hosting regions

Reference answer

No. Only EU (Paris/Frankfurt) and US (N. Virginia/Oregon) regions are offered.

Expected: Infrastructure hosting › Hosting regions

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly states that data cannot be hosted in Australia, matching the reference answer. Every statement in the answer is fully supported by the cited excerpts, and it covers all key points accurately.

Drafted answer

An accessibility conformance report (VPAT) is not yet available, although Vantrel is working toward conformance with WCAG 2.1 level AA.

• ↳ Product FAQ › Is Vantrel Plan accessible to users with disabilities?

Reference answer

Not yet. Vantrel is working toward WCAG 2.1 AA, but a VPAT is not available yet.

Expected: Product faq › Is Vantrel Plan accessible to users with disabilities?

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer perfectly matches the reference answer by stating that a VPAT is not yet available while noting that Vantrel is working toward WCAG 2.1 level AA. All statements are fully supported by the cited excerpts, and no key points from the reference are missing.

Drafted answer

The provided documentation does not contain information regarding FedRAMP authorization for Vantrel Plan.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation. The assistant correctly identified that the documentation does not contain this information and stated so, receiving top scores across all criteria.

Drafted answer

The provided documentation does not contain information on whether Vantrel will sign a HIPAA Business Associate Agreement (BAA).

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation, and the assistant correctly identified this and stated that the documentation does not contain the information. Therefore, all criteria are fully met with the maximum score.

Drafted answer

The provided documentation does not contain information about the coverage amount of our cyber insurance policy.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation, and the assistant correctly identified this and stated the information is missing. Therefore, it achieves maximum scores across all criteria.

Drafted answer

The provided documentation does not contain information regarding source code escrow agreements.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation. The assistant correctly identified that the documentation does not contain this information and did not cite any excerpts, matching the reference completely.

Drafted answer

The provided documentation does not contain information regarding IP allowlisting to restrict user access.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation, and the assistant correctly identified this and stated that the documentation does not contain the requested information. Therefore, all criteria receive the maximum score.

Drafted answer

The provided documentation does not contain information regarding the use of post-quantum cryptography.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation, and the assistant correctly identifies that the documentation does not contain this information without hallucinating any details.

Drafted answer

Vantrel provides a SOC 2 Type II report and an ISO 27001 certificate rather than a SOC 1 report.

• ↳ Data Processing Agreement (summary) › Audits
• ↳ Security Whitepaper › Certifications and audit reports

Reference answer

Not in the documentation. Only Vantrel's SOC 2 Type II is documented; the SOC 1 mentioned is AWS's, not Vantrel's.

Grader · correctness 1 · faithfulness 3 · completeness 1

The reference states that the information is not documented, meaning Vantrel's SOC 1 report is not mentioned. The assistant incorrectly answers that Vantrel provides a SOC 2 and ISO 27001 rather than a SOC 1, making it a definitive answer rather than stating the documentation lacks it. Therefore, correctness and completeness are low, but faithfulness is also affected since the excerpts do not state a SOC 1 is unavailable, only that SOC 2 and ISO 27001 are provided.

Drafted answer

The provided documentation does not contain information on the number of people who work in the security team.

Reference answer

Not in the documentation. Only the total headcount (~240) and the CISO/Security Committee are mentioned.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation. The assistant correctly identified that the information is missing, matching the reference conclusion.

Drafted answer

The documentation does not provide a mean time to resolve (MTTR) for P1 incidents. The response times for P1 incidents are first-response targets of 1 hour for Enterprise and 4 hours for Business customers, which are not resolution times.

• ↳ Service Level Agreement and Support › Support

Reference answer

Not in the documentation. Only first-response targets are documented, explicitly not resolution times.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identified that mean time to resolve (MTTR) is not documented, while accurately noting the documented first-response targets and their explicit caveat that they are not resolution times. The claims are fully supported by the cited excerpts, and it completely covers all aspects mentioned in the reference.

Drafted answer

The provided documentation does not contain information regarding CSA STAR certification.

Reference answer

Not in the documentation. The documented certifications are ISO 27001 and SOC 2 Type II.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that CSA STAR certification is not mentioned in the documentation, and the assistant correctly identified that the documentation does not contain this information without hallucinating or adding unsupported claims.

Drafted answer

The provided documentation does not contain information about an emergency phone number for security incidents.

Reference answer

Not in the documentation. Only email (security@vantrel.example) and the status page are documented.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation, which matches the assistant's response. The assistant correctly identifies that no phone number is provided, aligning with the reference, and since no information was found, faithfulness and completeness are also fully satisfied.

Drafted answer

There is a contradiction in the documentation regarding the deletion timeframe after contract termination. According to the Security Whitepaper (v2026.1), customer data is permanently deleted from production systems within 90 days of contract termination (S1). However, the Data Processing Agreement summary (v2025.3) states that data is deleted within 30 days after the end of the 30-day grace period (S2).

• ↳ Security Whitepaper › Data retention and deletion
• ↳ Data Processing Agreement (summary) › Return and deletion of data

Reference answer

The documentation is inconsistent: the Security Whitepaper says within 90 days of termination, while the DPA says within 30 days after a 30-day export grace period (i.e. ~60 days). Both sources should be cited and the inconsistency flagged.

Expected: Security whitepaper › Data retention and deletion · Data processing agreement › Return and deletion of data

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identified the contradiction between the Security Whitepaper (90 days) and the DPA (30 days after a 30-day grace period), cited both sources accurately, and fulfilled all criteria set by the reference answer.

Drafted answer

Pour les clients de la région EU (par défaut), les données sont hébergées sur AWS eu-west-3 (Paris, France), avec un emplacement de reprise après sinistre à Francfort, en Allemagne. Le stockage des données au repos, y compris les sauvegardes, reste dans la région sélectionnée.

• ↳ Infrastructure and Hosting › Hosting regions

Reference answer

Oui pour la région EU (par défaut) : AWS eu-west-3 à Paris, avec reprise d'activité à Francfort (Allemagne). Une région US existe aussi.

Expected: Infrastructure hosting › Hosting regions

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly matches the reference answer by confirming that data for the EU region is hosted in Paris, France, with disaster recovery in Frankfurt, and it is fully supported by the cited documentation excerpts, covering all key points accurately.

Drafted answer

Oui, Vantrel prend en charge l'authentification unique (SSO) avec Microsoft Entra ID via SAML 2.0 et OpenID Connect (OIDC) sur les plans Business et Enterprise.

• ↳ Identity and Access Management › Customer authentication

Reference answer

Oui, SSO SAML 2.0 et OIDC sur les plans Business et Enterprise ; Microsoft Entra ID fait partie des fournisseurs testés.

Expected: Identity access › Customer authentication

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly states that Vantrel supports SSO with Microsoft Entra ID via SAML 2.0 and OIDC on Business and Enterprise plans, matching the reference answer. Every statement in the answer is directly supported by the provided documentation excerpts. All key points from the reference are fully covered.

Drafted answer

The provided documentation does not contain information regarding SecNumCloud qualification by ANSSI.

Reference answer

Pas dans la documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not documented. The assistant correctly identifies that the documentation does not contain this information and states so, which earns top scores across all criteria.