Drafted by Mistral, graded by Gemini

Drafted answer

Yes, customer data stored in databases, file storage, and backups is encrypted at rest using **AES-256**. Keys are managed by Vantrel in **AWS Key Management Service (KMS)** and rotated automatically every year. **Customer-managed keys (BYOK)** are supported for Enterprise plan customers, allowing them to provide and control their own encryption keys via AWS KMS.

• ↳ Security Whitepaper › Encryption

Reference answer

Yes. All customer data (databases, file storage, backups) is encrypted at rest with AES-256, keys managed in AWS KMS and rotated automatically every year.

Expected: Security whitepaper › Encryption

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly states that customer data is encrypted at rest using AES-256 with AWS KMS keys rotated annually, matching the reference answer. It also correctly includes additional details about customer-managed keys (BYOK) which are fully supported by the provided documentation excerpts. All statements are entirely faithful to the cited excerpts and complete.

Drafted answer

The minimum TLS version supported for data in transit is **TLS 1.2**. TLS 1.3 is preferred and negotiated when supported by the client, while older protocols (SSL, TLS 1.0, and TLS 1.1) are disabled.

• ↳ Security Whitepaper › Encryption

Reference answer

TLS 1.2 is the minimum; TLS 1.3 is preferred. SSL, TLS 1.0 and 1.1 are disabled, and HSTS is enforced.

Expected: Security whitepaper › Encryption

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies TLS 1.2 as the minimum version, mentions that TLS 1.3 is preferred, and notes that older protocols are disabled, matching the reference answer completely. Every statement is directly supported by the cited documentation excerpt, and all key points from the reference are included.

Drafted answer

Yes, Vantrel has been **ISO/IEC 27001:2022 certified since September 2023**, covering the development, operation, and support of Vantrel Plan. Annual surveillance audits are conducted to maintain compliance.

• ↳ Security Whitepaper › Certifications and audit reports

Reference answer

Yes, ISO/IEC 27001:2022 certified since September 2023, covering development, operation and support of Vantrel Plan, with annual surveillance audits.

Expected: Security whitepaper › Certifications and audit reports

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer perfectly matches the reference answer in terms of facts, covering the certification standard, the exact date, the scope, and the annual surveillance audits. Every single statement is directly supported by the cited security whitepaper excerpt. All key points from the reference are fully addressed.

Drafted answer

Yes, Vantrel can provide the SOC 2 Type II report under a non-disclosure agreement (NDA) upon request to **security@vantrel.example**. The report covers the Security, Availability, and Confidentiality Trust Services Criteria and is typically available by March of the year following the audit period (January 1–December 31).

• ↳ Security Whitepaper › Certifications and audit reports

Reference answer

Yes. Vantrel has an annual SOC 2 Type II (Security, Availability, Confidentiality; period Jan 1 - Dec 31, report available around March). It is shared under NDA on request to security@vantrel.example.

Expected: Security whitepaper › Certifications and audit reports

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly matches all facts provided in the reference answer regarding the SOC 2 Type II report availability, scope, and contact method. Every statement made by the assistant is directly supported by the cited documentation excerpts, and all key points from the reference are fully covered.

Drafted answer

Customer data for European customers is hosted in the European Union, specifically in **AWS eu-west-3 (Paris, France)** as the primary location, with disaster recovery in **AWS eu-central-1 (Frankfurt, Germany)**. Data at rest, including backups and disaster recovery copies, remains within the EU region. Limited remote access by support staff outside the EU is covered by the **European Commission's Standard Contractual Clauses (SCCs)** and a transfer impact assessment.

• ↳ Data Processing Agreement (summary) › International data transfers (not in expected sources)
• ↳ Infrastructure and Hosting › Hosting regions

Reference answer

On AWS in eu-west-3 (Paris, France), with disaster recovery in eu-central-1 (Frankfurt, Germany). Data at rest, including backups, stays in the selected region.

Expected: Infrastructure hosting › Hosting regions

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer accurately captures the primary hosting location and disaster recovery region from the documentation, matching the reference answer well. Every statement made by the assistant is directly supported by the cited excerpts, including the additional details on remote access and SCCs. Furthermore, all key points of the reference are fully covered.

Drafted answer

The **Recovery Point Objective (RPO)** is **1 hour**, and the **Recovery Time Objective (RTO)** is **4 hours** for a major incident affecting an entire AWS region, as documented in the Business Continuity and Disaster Recovery Plan (v2026.1).

• ↳ Business Continuity and Disaster Recovery › Objectives

Reference answer

RPO of 1 hour and RTO of 4 hours.

Expected: Business continuity › Objectives

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identified the RPO as 1 hour and RTO as 4 hours, perfectly matching the reference answer. Every statement in the answer is fully supported by the cited excerpts from the Business Continuity and Disaster Recovery Plan. All key points are covered accurately and completely.

Drafted answer

Backups are taken continuously with point-in-time recovery available for the last 7 days, and full encrypted snapshots are taken daily. These backups are retained for 35 days.

• ↳ Business Continuity and Disaster Recovery › Backups

Reference answer

Continuous backups with 7-day point-in-time recovery, plus daily encrypted full snapshots retained for 35 days, replicated to the DR location. Restores are tested monthly.

Expected: Business continuity › Backups

Grader · correctness 4 · faithfulness 5 · completeness 4

The assistant's answer covers the frequency and retention of backups accurately and is fully supported by the cited documentation. However, it misses a few minor points from the reference answer such as the replication to the DR location and the monthly restoration tests, leading to a slightly lower completeness score.

Drafted answer

Vantrel Plan supports **SAML 2.0** and **OpenID Connect (OIDC)** for Single Sign-On (SSO), available only on the **Business and Enterprise plans**.

• ↳ Identity and Access Management › Customer authentication

Reference answer

SAML 2.0 and OpenID Connect, on Business and Enterprise plans. Tested with Microsoft Entra ID, Okta and Google Workspace. Enforcing SSO disables password login.

Expected: Identity access › Customer authentication

Grader · correctness 4 · faithfulness 5 · completeness 4

The assistant correctly identifies SAML 2.0 and OpenID Connect on Business and Enterprise plans, matching the reference facts well, though it misses the minor detail about tested IdPs and password disabling which are present in the excerpts and reference. All statements made are fully supported by the cited text. It covers the core requirements of the question.

Drafted answer

SCIM 2.0 for automated user provisioning and deprovisioning is supported **only on the Enterprise plan**. Other plans require manual user management by workspace administrators.

• ↳ Identity and Access Management › User provisioning

Reference answer

Yes, SCIM 2.0 is supported, on the Enterprise plan only.

Expected: Identity access › User provisioning

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer perfectly matches the reference answer, accurately reflecting both the support for SCIM 2.0 and the restriction to the Enterprise plan. Every statement is directly supported by the cited documentation excerpt, and all key points from the reference are fully covered.

Drafted answer

For Enterprise customers, we commit to a **monthly uptime of 99.95%**, excluding scheduled maintenance. Service credits are applied if uptime falls below this threshold.

• ↳ Service Level Agreement and Support › Availability commitment

Reference answer

99.95% monthly uptime for Enterprise (excluding scheduled maintenance), with service credits of 10% or 25% of the monthly fee.

Expected: Sla support › Availability commitment

Grader · correctness 4 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies the 99.95% monthly uptime commitment for Enterprise customers and mentions service credits generally, though it leaves out the specific credit percentages (10% or 25%) found in the reference answer. Every statement is fully supported by the cited excerpts.

Drafted answer

Vantrel will notify you of a personal data breach without undue delay, and in any case within **48 hours** after becoming aware of it. The notification includes details on the nature of the breach, affected data categories, likely consequences, and measures taken.

• ↳ Data Processing Agreement (summary) › Personal data breach notification

Reference answer

Without undue delay and in any case within 48 hours after becoming aware of it.

Expected: Data processing agreement › Personal data breach notification

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer perfectly matches the reference answer by stating the 48-hour timeframe and notifying without undue delay. Every statement is fully supported by the cited excerpt from the Data Processing Agreement, and all key points are covered accurately.

Drafted answer

Vantrel performs penetration testing at least once a year, with additional tests conducted before major architectural changes. The most recent penetration test was completed in March 2026 and identified no critical findings, along with two high-severity findings that were remediated within 30 days and verified during a retest.

• ↳ Penetration Testing and Vulnerability Management › Penetration testing

Reference answer

At least annually by an independent third party, plus before major architectural changes. The last test (March 2026) found no critical and two high findings, both remediated within 30 days and verified by retest.

Expected: Vulnerability management › Penetration testing

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer perfectly matches the reference answer in all aspects, including the frequency of penetration tests, the details of the last test in March 2026, and the remediation of findings. All statements are fully supported by the provided documentation excerpts.

Drafted answer

Critical vulnerabilities must be remediated within **7 days** as per Vantrel’s vulnerability management policy.

• ↳ Penetration Testing and Vulnerability Management › Vulnerability management

Reference answer

Critical vulnerabilities must be remediated within 7 days (high: 30 days, medium: 90 days).

Expected: Vulnerability management › Vulnerability management

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identified the remediation timeline for critical vulnerabilities as 7 days, which matches the reference answer and is fully supported by the cited documentation excerpts. All key points are covered accurately and faithfully without any hallucination.

Drafted answer

Vantrel’s sub-processors are categorized as follows: **Infrastructure sub-processors** include Amazon Web Services EMEA SARL (cloud hosting, storage, backups), Cloudflare, Inc. (CDN, Web Application Firewall, DDoS protection), and Datadog, Inc. (infrastructure and application monitoring, logs). **Service sub-processors** include Stripe Payments Europe, Ltd. (billing and subscription payments), Zendesk, Inc. (customer support ticketing), SendGrid (Twilio Inc.) (transactional emails), and Microsoft Ireland Operations Ltd. (Azure OpenAI Service, for AI features like Vantrel Assist).

• ↳ Sub-processors › Infrastructure sub-processors
• ↳ Sub-processors › Service sub-processors
• ↳ Sub-processors › Notes (not in expected sources)

Reference answer

AWS (hosting), Cloudflare (CDN/WAF/DDoS), Datadog (monitoring), Stripe (billing), Zendesk (support), SendGrid/Twilio (transactional emails), Microsoft Azure OpenAI Service (AI features).

Expected: Subprocessors › Infrastructure sub-processors · Subprocessors › Service sub-processors

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer completely and accurately matches the reference answer by listing all sub-processors and their respective purposes. Every single statement is fully supported by the provided documentation excerpts.

Drafted answer

No, customer data is **never used to train or fine-tune** AI models by Vantrel or its AI sub-processors. Only the data required to answer a specific request (e.g., selected cells and their labels) is sent to the model, and prompts/responses are not stored beyond the user's session, except for audit logging of AI feature usage (without content).

• ↳ AI Features Policy › Use of customer data

Reference answer

No. Customer data is never used to train or fine-tune AI models, neither by Vantrel nor by its AI sub-processor.

Expected: Ai features policy › Use of customer data

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly matches the reference answer, stating that customer data is never used to train or fine-tune AI models. Every statement made by the assistant is directly supported by the cited excerpts, and all key points from the reference are fully covered.

Drafted answer

Yes, your employees can sign in with their corporate Okta accounts using **Single Sign-On (SSO)** via **OpenID Connect (OIDC)** or **SAML 2.0**, provided your organization is on the **Business or Enterprise plan**. Okta is explicitly listed as a tested identity provider.

• ↳ Identity and Access Management › Customer authentication

Reference answer

Yes, via SAML 2.0 or OIDC SSO on Business and Enterprise plans; Okta is a tested identity provider.

Expected: Identity access › Customer authentication

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer accurately matches the reference answer by confirming that employees can use Okta via SAML 2.0 or OIDC on Business and Enterprise plans, and notes that Okta is a tested identity provider. All statements are fully supported by the cited documentation excerpt, and all key points from the reference are covered.

Drafted answer

In the event of a major incident affecting the primary AWS region (e.g., Paris, France), up to **1 hour of data** could be lost, as the documented **Recovery Point Objective (RPO)** is **1 hour** (S5). This aligns with continuous database backups and point-in-time recovery for the last 7 days (S2).

• ↳ Business Continuity and Disaster Recovery › Backups (not in expected sources)
• ↳ Business Continuity and Disaster Recovery › Objectives

Reference answer

At most about 1 hour of data (RPO 1 hour), with service restored from the Frankfurt DR region within 4 hours (RTO).

Expected: Business continuity › Objectives · Infrastructure hosting › Hosting regions

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies the RPO as 1 hour, aligning with the reference answer. All statements are fully faithful to the cited excerpts, and the key points regarding data loss (RPO) are completely covered.

Drafted answer

No, Vantrel engineers do not have standing access to customer data, including budget figures. Access to customer workspaces, including sensitive data, requires explicit and temporary customer approval via the administration console, with a maximum duration of 72 hours.

• ↳ Identity and Access Management › Vantrel employee access to customer data

Reference answer

No standing access. Support can only access a workspace when the customer grants temporary access (max 72 hours), which is logged and visible in the audit trail.

Expected: Identity access › Vantrel employee access to customer data

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly states that engineers do not have standing access and can only access workspaces with temporary customer approval (max 72 hours), which matches the reference answer. Every statement in the answer is fully supported by the cited documentation excerpt. Finally, the answer covers all key points of the reference without missing any information.

Drafted answer

Yes, Enterprise customers can provide and control their own encryption keys for data at rest through AWS Key Management Service (KMS) under a **Customer-Managed Keys (BYOK)** model. Revoking the key renders the customer's data unreadable by Vantrel.

• ↳ Security Whitepaper › Encryption

Reference answer

Yes, Enterprise customers can use customer-managed keys (BYOK) via AWS KMS; revoking the key makes the data unreadable by Vantrel.

Expected: Security whitepaper › Encryption

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer perfectly matches the reference answer in facts, captures all key points, and every statement is directly supported by the cited documentation excerpt.

Drafted answer

Yes, Vantrel performs background checks on all new employees before their start date, where permitted by local law.

• ↳ Security Whitepaper › Employee security

Reference answer

Yes, background checks are performed on all new employees before their start date, where permitted by local law.

Expected: Security whitepaper › Employee security

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer directly matches the reference answer, stating that background checks are performed on all new employees before their start date where permitted by local law. Every statement in the answer is fully supported by the cited excerpt, and all key points from the reference are covered.

Drafted answer

For **Business plan** customers, a first response to a service outage (P1) is targeted within **4 hours during business hours (Mon–Fri)**. For **Enterprise plan** customers, a first response is targeted within **1 hour, 24/7**, including at 3am. **Starter plan** customers receive best-effort email support only, with no guaranteed response time.

• ↳ Service Level Agreement and Support › Support

Reference answer

Enterprise: P1 first response within 1 hour, 24/7. Business: support is business hours only, P1 first response within 4 hours.

Expected: Sla support › Support

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly details the response times for both Enterprise and Business plans at 3 am, aligning with the reference answer. The claims are fully supported by the cited excerpts from the support documentation, covering all necessary details comprehensively.

Drafted answer

Yes, several sub-processors process customer data outside the EU. Stripe Payments Europe, Ltd. processes billing and subscription payment data in the **US**, SendGrid (Twilio Inc.) processes transactional emails in the **US**, and Datadog, Inc. processes infrastructure and application monitoring, logs in the **US** for customers hosted in the US region. Additionally, AWS EMEA SARL may process data in the **US** if the customer selects the US region for hosting, storage, and backups.

• ↳ Sub-processors › Service sub-processors
• ↳ Sub-processors › Infrastructure sub-processors
• ↳ Sub-processors › Notes (not in expected sources)

Reference answer

For EU tenants, hosting (AWS), monitoring (Datadog), support (Zendesk) and AI (Azure EU Data Zone) stay in the EU. Cloudflare handles traffic in transit on its global network, Stripe operates in EU/US, and SendGrid processes email addresses and notification content in the US. Transfers are covered by SCCs; Vantrel Inc. is under the EU-U.S. Data Privacy Framework.

Expected: Subprocessors › Infrastructure sub-processors · Subprocessors › Service sub-processors · Data processing agreement › International data transfers

Grader · correctness 3 · faithfulness 2 · completeness 4

The assistant's answer incorrectly states that Datadog and AWS process EU data in the US, whereas the documentation and reference clarify that EU tenants keep their monitoring and hosting in the EU. Furthermore, the assistant's claims about Datadog and AWS processing EU customer data in the US contradict the provided excerpts, making the answer unfaithful and incorrect for an EU tenant context.

Drafted answer

Customers are informed of security incidents involving their data via email notification by the Security team, in accordance with timelines defined in the Data Processing Agreement. For **personal data breaches**, customers are notified **without undue delay and within 48 hours** of Vantrel becoming aware of the breach, including details such as the nature of the breach, affected data categories, and remediation measures (S4). For **SEV1 incidents** (e.g., confirmed data breaches or complete service outages), updates are also published on the public status page (status.vantrel.example) with updates at least every 60 minutes during the incident (S1). A post-incident report is shared with affected customers within **10 business days** after resolution (S1).

• ↳ Incident Response › Customer communication
• ↳ Data Processing Agreement (summary) › Personal data breach notification

Reference answer

Availability incidents on the status page (updates at least every 60 min for SEV1); customers whose data is affected are emailed by Security within the DPA timeline (48 hours); a root cause analysis is shared within 10 business days after a SEV1.

Expected: Incident response › Customer communication · Data processing agreement › Personal data breach notification

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer covers all key points present in the reference answer, including status page updates for SEV1 incidents, email notifications within the DPA/48-hour timeline, and the 10-business-day timeline for root cause analysis. Every statement is fully supported by the cited excerpts, and the facts are completely accurate and consistent with the reference.

Drafted answer

For **customer users**, Vantrel supports **TOTP-based MFA** (authenticator app) and **FIDO2/WebAuthn security keys**. SMS-based MFA is **not supported**. Administrators can enforce MFA for all users in a workspace. For **Vantrel staff accessing production**, the documentation does not explicitly describe MFA requirements, but access to production environments is restricted to the **Platform Engineering and Security teams** (S3), implying controlled, role-based access. No MFA specifics for Vantrel staff are provided in the excerpts.

• ↳ Identity and Access Management › Customer authentication
• ↳ Infrastructure and Hosting › Environments (not in expected sources)

Reference answer

Customers: TOTP or FIDO2/WebAuthn, can be made mandatory by admins, no SMS. Staff: production admin access goes through a bastion with SSO, MFA and session recording; direct SSH disabled.

Expected: Identity access › Customer authentication · Infrastructure hosting › Network security

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant correctly answered the customer MFA options using the documentation, but failed to mention the staff production access details (SSO, MFA, session recording, bastion host) which were present in the source documentation, though it claimed they were not documented.

Drafted answer

Infrastructure and application logs are centralized in **Datadog**, with EU-hosted tenants using the EU site. Logs are retained for **12 months**. Audit logs (covering user activity, permissions, model changes, and data exports) are retained for **13 months** and can be exported via API or viewed in the admin console. Enterprise customers can additionally stream audit logs to their SIEM via webhook.

• ↳ Infrastructure and Hosting › Logging and monitoring
• ↳ Identity and Access Management › Audit logs

Reference answer

Infrastructure and application logs are centralized in Datadog (EU site for EU tenants), retained 12 months. Customer audit logs are retained 13 months and exportable via API or streamed to a SIEM (Enterprise).

Expected: Infrastructure hosting › Logging and monitoring · Identity access › Audit logs

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer perfectly matches the reference answer in terms of facts, covering Datadog centralization, EU tenant site routing, 12-month infrastructure log retention, and 13-month audit log retention with export capabilities. Every statement made by the assistant is fully supported by the provided excerpts, and all key points of the reference are comprehensively addressed.

Drafted answer

Yes, when **Vantrel Assist** is enabled, customer data is sent to **Microsoft Ireland Operations Ltd. (Azure OpenAI Service)** in the **EU Data Zone** for processing by the AI model. Only the data required to answer a specific request (e.g., selected cells and their labels) is transmitted, and it is not used for training or fine-tuning. Data is not sent to other third-party sub-processors for AI purposes.

• ↳ AI Features Policy › Use of customer data
• ↳ Sub-processors › Service sub-processors
• ↳ AI Features Policy › Vantrel Assist
• ↳ Sub-processors › Notes (not in expected sources)

Reference answer

Only if a workspace Admin enables Vantrel Assist (off by default). Then only the data needed for the request is sent to Microsoft Azure OpenAI Service in the EU Data Zone; it is never used for training.

Expected: Ai features policy › Vantrel Assist · Ai features policy › Use of customer data · Subprocessors › Service sub-processors

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that data is sent to Microsoft Azure OpenAI Service in the EU Data Zone only when Vantrel Assist is enabled, covering all key points of the reference answer. Every statement made by the assistant is fully supported by the provided documentation excerpts.

Drafted answer

Vantrel operates a **private bug bounty program** with invited researchers, but the **reward amounts are not specified** in the provided documentation.

• ↳ Incident Response › Reporting a vulnerability

Reference answer

Yes, a private bug bounty program with invited researchers. Reward amounts are not documented.

Expected: Incident response › Reporting a vulnerability

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identified that the company operates a private bug bounty program with invited researchers and noted that reward amounts are not specified. All facts are fully supported by the cited excerpt and match the reference answer completely.

Drafted answer

The documentation confirms that Vantrel's most recent penetration test was conducted by an **independent third-party security firm**, but the **specific name of the firm is not disclosed** in the provided excerpts.

• ↳ Penetration Testing and Vulnerability Management › Penetration testing

Reference answer

An independent third-party security firm, in March 2026. The firm's name is not documented.

Expected: Vulnerability management › Penetration testing

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identified that the test was done by an independent third-party and that the specific name is not mentioned, which matches the reference answer. All statements are fully supported by the provided excerpts and cover all key points.

Drafted answer

Audit logs can be streamed to a SIEM via webhook for **Enterprise plan customers only**, but the documentation does **not specify** which SIEM products are supported. The webhook integration is explicitly mentioned for streaming audit logs to a customer's SIEM (S1).

• ↳ Identity and Access Management › Audit logs

Reference answer

Enterprise customers can stream audit logs to their SIEM via webhook; specific supported SIEM products are not listed.

Expected: Identity access › Audit logs

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identified that specific SIEM products are not listed in the documentation while mentioning the webhook streaming option for Enterprise customers. The answer is fully faithful to the provided excerpt and captures all key points of the reference answer.

Drafted answer

The Data Protection Officer (DPO) at Vantrel can be contacted at **dpo@vantrel.example**.

• ↳ Company Overview › Security and compliance organization

Reference answer

The DPO can be contacted at dpo@vantrel.example; the DPO's name is not documented.

Expected: Company overview › Security and compliance organization

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies the DPO contact email and acknowledges it based on the excerpts. However, it does not explicitly state that the DPO's name is missing, though the reference notes the name is not documented. Overall, the core available facts are presented accurately and faithfully supported by the citations.

Drafted answer

No, custom roles are not currently supported. Vantrel Plan uses predefined role-based access control (RBAC) with four default roles: Admin, Modeler, Contributor, and Viewer.

• ↳ Identity and Access Management › Roles and permissions

Reference answer

No, custom roles are not currently supported. Four default roles exist (Admin, Modeler, Contributor, Viewer) plus dimension-member-level restrictions.

Expected: Identity access › Roles and permissions

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly answered that custom roles are not supported, matching the reference answer. All statements are fully supported by the cited documentation excerpts, and all key points regarding the default roles are included.

Drafted answer

No, Vantrel Plan is delivered exclusively as a multi-tenant Software-as-a-Service (SaaS) application and does not offer on-premise, private cloud, or self-hosted deployment options.

• ↳ Company Overview › Product and delivery model

Reference answer

No. Vantrel Plan is SaaS only; no on-premise, private cloud or self-hosted deployment.

Expected: Company overview › Product and delivery model

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly states that on-premise deployment is not offered, aligning perfectly with the reference answer. All statements in the answer are directly supported by the provided documentation excerpt. The response is fully complete and covers all key points of the reference.

Drafted answer

No, SMS-based multi-factor authentication (MFA) is not supported for customer users in Vantrel Plan.

• ↳ Identity and Access Management › Customer authentication

Reference answer

No, SMS-based MFA is not supported (TOTP and FIDO2/WebAuthn are).

Expected: Identity access › Customer authentication

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly states that SMS-based MFA is not supported, perfectly matching the reference answer. Every statement in the answer is directly supported by the cited documentation excerpt, and all key points are covered accurately.

Drafted answer

No, Vantrel does not currently offer hosting in Australia. Only EU and US regions are available as of the provided documentation (v2026.1).

• ↳ Infrastructure and Hosting › Hosting regions

Reference answer

No. Only EU (Paris/Frankfurt) and US (N. Virginia/Oregon) regions are offered.

Expected: Infrastructure hosting › Hosting regions

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly answered 'No' based on the reference answer and the provided excerpts, which only list EU and US regions. Every statement in the answer is directly supported by the cited documentation, and all key points regarding the lack of Australian hosting are fully addressed.

Drafted answer

No, Vantrel does not currently have an accessibility conformance report (VPAT). The company is working toward conformance with WCAG 2.1 level AA, but no VPAT is available as of the latest documentation (v2026.3).

• ↳ Product FAQ › Is Vantrel Plan accessible to users with disabilities?

Reference answer

Not yet. Vantrel is working toward WCAG 2.1 AA, but a VPAT is not available yet.

Expected: Product faq › Is Vantrel Plan accessible to users with disabilities?

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly states that a VPAT is not yet available and mentions the work toward WCAG 2.1 AA, perfectly matching the reference answer. Every statement is directly supported by the cited excerpt, and all key points of the reference are covered.

Drafted answer

The documentation does not mention whether Vantrel Plan is FedRAMP authorized.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation. The assistant correctly identifies that the documentation does not mention FedRAMP authorization for Vantrel Plan, adhering to the expected outcome.

Drafted answer

The documentation does not specify whether Vantrel will sign a HIPAA Business Associate Agreement (BAA).

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation, and the assistant correctly states that the documentation does not specify this. Therefore, all criteria are fully met with the maximum score.

Drafted answer

The documentation excerpts provided do not mention the coverage amount of Vantrel's cyber insurance policy.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation, and the assistant correctly identified this as well. Since no information was available, faithfulness and completeness are also fully satisfied by the assistant's correct acknowledgment of the omission.

Drafted answer

The provided documentation does not mention whether Vantrel offers a source code escrow agreement.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation. The assistant correctly identifies that the documentation does not mention source code escrow agreements, yielding top scores across all criteria.

Drafted answer

The documentation does not specify whether IP allowlisting is supported to restrict user access.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation, and the assistant correctly identifies that the documentation does not specify this, citing no excerpts as expected.

Drafted answer

The documentation does not specify whether Vantrel uses post-quantum cryptography.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation. The assistant correctly identifies that the documentation does not specify this information, and since it correctly notes the lack of documentation, it receives top scores across all criteria.

Drafted answer

The documentation does not mention whether a SOC 1 report is available or provided to customers.

Reference answer

Not in the documentation. Only Vantrel's SOC 2 Type II is documented; the SOC 1 mentioned is AWS's, not Vantrel's.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation, but notes that only a SOC 2 Type II is documented while the SOC 1 belongs to AWS. The assistant correctly recognized that the SOC 1 report is not mentioned in relation to Vantrel, aligning with the reference.

Drafted answer

The documentation does not specify the number of people working in the security team.

Reference answer

Not in the documentation. Only the total headcount (~240) and the CISO/Security Committee are mentioned.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation, and the assistant correctly identifies that the documentation does not specify the number of people in the security team. Since the information is absent and both the reference and assistant agree on this point, all criteria are scored the maximum.

Drafted answer

The documentation specifies **first-response targets** for P1 incidents (service unavailable) as **4 hours for Business plan customers** and **1 hour for Enterprise plan customers** during business hours (Mon–Fri) or 24/7, respectively (S1). However, the documentation does **not** provide explicit details on **mean time to resolve (MTTR)** P1 incidents, only first-response targets.

• ↳ Service Level Agreement and Support › Support

Reference answer

Not in the documentation. Only first-response targets are documented, explicitly not resolution times.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identified that the documentation does not contain mean time to resolve (MTTR) for P1 incidents, aligning with the reference answer. It also correctly cited the first-response targets from the documentation, and every statement made is fully supported by the cited excerpts.

Drafted answer

The documentation does not mention CSA STAR certification.

Reference answer

Not in the documentation. The documented certifications are ISO 27001 and SOC 2 Type II.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that CSA STAR is not in the documentation, and the assistant correctly identified that the documentation does not mention it. Since the information is correctly noted as absent without any hallucinations, all criteria receive the maximum score.

Drafted answer

The documentation does not specify an emergency phone number for security incidents.

Reference answer

Not in the documentation. Only email (security@vantrel.example) and the status page are documented.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation, and the assistant correctly identifies that the documentation does not specify an emergency phone number. Since no information was found, faithfulness and completeness are also fully satisfied by stating the absence of the information.

Drafted answer

The documentation provides conflicting information on the data deletion timeline after contract termination. According to the **Security Whitepaper (v2026.1)**, customer data is permanently deleted from production systems within **90 days** of contract termination (S1). However, the **Data Processing Agreement (summary) (v2025.3)** states that deletion occurs within **30 days after the end of a 30-day grace period** (S2), implying a **60-day total window** (30 days grace + 30 days deletion). The discrepancy arises between the two sources.

• ↳ Security Whitepaper › Data retention and deletion
• ↳ Data Processing Agreement (summary) › Return and deletion of data

Reference answer

The documentation is inconsistent: the Security Whitepaper says within 90 days of termination, while the DPA says within 30 days after a 30-day export grace period (i.e. ~60 days). Both sources should be cited and the inconsistency flagged.

Expected: Security whitepaper › Data retention and deletion · Data processing agreement › Return and deletion of data

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identified the conflicting information between the Security Whitepaper (90 days) and the DPA (30-day grace period plus 30 days for deletion), citing both sources and properly flagging the inconsistency as required by the reference answer.

Drafted answer

Les données des clients hébergés dans la région **EU** sont stockées dans **AWS eu-west-3 (Paris, France)** en tant que **emplacement principal**, mais des copies de sauvegarde et de reprise après sinistre sont également stockées dans **AWS eu-central-1 (Francfort, Allemagne)**. Aucune information n'est fournie sur la répartition exacte des données entre ces deux emplacements ou sur une éventuelle restriction géographique stricte à la France.

• ↳ Infrastructure and Hosting › Hosting regions

Reference answer

Oui pour la région EU (par défaut) : AWS eu-west-3 à Paris, avec reprise d'activité à Francfort (Allemagne). Une région US existe aussi.

Expected: Infrastructure hosting › Hosting regions

Grader · correctness 3 · faithfulness 3 · completeness 3

The assistant correctly states that data for the EU region is hosted in Paris, but adds an incorrect caveat that data is also stored in Frankfurt for disaster recovery when the excerpt explicitly states that customer data stays within the selected region. The faithfulness is also impacted by this incorrect interpretation of the backup location, and completeness is partially affected by missing the mention of the US region.

Drafted answer

Oui, Vantrel Plan propose l’authentification unique (SSO) avec Microsoft Entra ID via **SAML 2.0** et **OpenID Connect (OIDC)**, mais uniquement sur les plans **Business** et **Enterprise**.

• ↳ Identity and Access Management › Customer authentication

Reference answer

Oui, SSO SAML 2.0 et OIDC sur les plans Business et Enterprise ; Microsoft Entra ID fait partie des fournisseurs testés.

Expected: Identity access › Customer authentication

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer perfectly matches the reference answer in terms of correctness and completeness, confirming SSO with Microsoft Entra ID via SAML 2.0 and OIDC on Business and Enterprise plans. Furthermore, every statement made in the answer is fully supported by the cited documentation excerpt, ensuring high faithfulness.

Drafted answer

Les excerpts fournis ne mentionnent pas la qualification SecNumCloud par l'ANSSI.

Reference answer

Pas dans la documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The reference answer states that the information is not in the documentation. The assistant correctly identified that the provided excerpts do not mention SecNumCloud qualification, which aligns with the reference.