Drafted by Gemini, graded by Mistral

Drafted answer

Yes, all customer data stored in databases, file storage, and backups is encrypted at rest using AES-256. Keys are managed in AWS Key Management Service (KMS) and are rotated automatically every year. Additionally, Enterprise customers have the option to provide and control their own encryption key through AWS KMS (BYOK).

• ↳ Security Whitepaper › Encryption

Reference answer

Yes. All customer data (databases, file storage, backups) is encrypted at rest with AES-256, keys managed in AWS KMS and rotated automatically every year.

Expected: Security whitepaper › Encryption

Grader · correctness 3 · faithfulness 3 · completeness 5

The assistant's answer is mostly correct and aligns with the reference answer for the core points: encryption of customer data at rest (databases, file storage, backups) using AES-256, key management in AWS KMS, and automatic yearly key rotation. However, the assistant added an unsupported detail about **Enterprise customers having the option to provide and control their own encryption key (BYOK)** without explicitly citing the relevant excerpt in the reasoning or answer. While the BYOK excerpt exists, the assistant did not clearly link it to the question about key management, making the inclusion of this detail appear as an unsupported extrapolation rather than a direct citation. The reference answer does not mention BYOK, so this is an unnecessary addition that slightly reduces faithfulness and correctness scores. Completeness is fully met since all key points from the reference are covered, and the assistant did not omit anything critical.

Drafted answer

All connections to Vantrel Plan use TLS 1.2 or higher, with TLS 1.3 preferred and negotiated whenever supported by the client. Older protocols such as SSL, TLS 1.0, and 1.1 are disabled.

• ↳ Security Whitepaper › Encryption

Reference answer

TLS 1.2 is the minimum; TLS 1.3 is preferred. SSL, TLS 1.0 and 1.1 are disabled, and HSTS is enforced.

Expected: Security whitepaper › Encryption

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer aligns closely with the reference answer, correctly stating that TLS 1.2 is the minimum supported version, TLS 1.3 is preferred, and older protocols (SSL, TLS 1.0, and 1.1) are disabled. The assistant also accurately includes the enforcement of HSTS, which is a key point in the reference. The cited excerpts fully support all claims made in the answer, and no additional unsupported details are included. The answer is complete, covering all key points from the reference without omissions or extraneous information.

Drafted answer

Yes, Vantrel has been certified against ISO/IEC 27001:2022 since September 2023. The certificate covers the development, operation, and support of Vantrel Plan.

• ↳ Security Whitepaper › Certifications and audit reports

Reference answer

Yes, ISO/IEC 27001:2022 certified since September 2023, covering development, operation and support of Vantrel Plan, with annual surveillance audits.

Expected: Security whitepaper › Certifications and audit reports

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer is **correct** as it aligns perfectly with the reference answer, including the certification date (September 2023) and the scope (development, operation, and support of Vantrel Plan). The **faithfulness** is also high, as all claims are directly supported by the cited excerpt from the Security Whitepaper. However, the **completeness** is slightly reduced because the assistant omitted the mention of *annual surveillance audits*, which is a key detail in the reference answer. The assistant’s answer is otherwise accurate and well-supported, with no hallucinations or unsupported claims.

Drafted answer

Yes, Vantrel undergoes an annual SOC 2 Type II examination covering Security, Availability, and Confidentiality Trust Services Criteria, and the report can be shared with customers and prospects under a non-disclosure agreement (NDA) upon request to security@vantrel.example.

• ↳ Security Whitepaper › Certifications and audit reports

Reference answer

Yes. Vantrel has an annual SOC 2 Type II (Security, Availability, Confidentiality; period Jan 1 - Dec 31, report available around March). It is shared under NDA on request to security@vantrel.example.

Expected: Security whitepaper › Certifications and audit reports

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is factually consistent with the reference answer, correctly stating that Vantrel provides a SOC 2 Type II report covering Security, Availability, and Confidentiality, and that it is shared under NDA upon request. The cited excerpts fully support these claims, including the audit period (Jan 1 - Dec 31) and the timing of report availability (March). The assistant’s answer is also complete, covering all key points from the reference (availability, scope, and sharing conditions). No contradictions or omissions are present, and the assistant did not introduce unsupported details or hallucinations.

Drafted answer

For European customers, customer data is hosted in the EU region (AWS eu-west-3 in Paris, France), with disaster recovery copies in AWS eu-central-1 (Frankfurt, Germany).

• ↳ Infrastructure and Hosting › Hosting regions

Reference answer

On AWS in eu-west-3 (Paris, France), with disaster recovery in eu-central-1 (Frankfurt, Germany). Data at rest, including backups, stays in the selected region.

Expected: Infrastructure hosting › Hosting regions

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies the primary hosting region (AWS eu-west-3 in Paris) and disaster recovery location (AWS eu-central-1 in Frankfurt) for European customers, matching the reference answer. However, the assistant's answer omits the critical detail about **data at rest** (including backups) staying within the selected region, which is explicitly stated in the reference and excerpts. The assistant's response is otherwise faithful to the cited documentation and covers the primary locations accurately, though it misses a key operational constraint (data residency).

Drafted answer

For Vantrel Plan, the Recovery Point Objective (RPO) is 1 hour and the Recovery Time Objective (RTO) is 4 hours.

• ↳ Business Continuity and Disaster Recovery › Objectives

Reference answer

RPO of 1 hour and RTO of 4 hours.

Expected: Business continuity › Objectives

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly identifies the RPO and RTO for the Vantrel Plan as 1 hour and 4 hours, respectively, which aligns perfectly with the reference answer. The cited excerpts explicitly state these values, ensuring both faithfulness and correctness. The assistant also correctly references the Vantrel Plan, which is a key detail in the question context. The answer is complete as it covers all key points from the reference without omissions or unsupported claims.

Drafted answer

Databases are backed up continuously with point-in-time recovery available for the last 7 days, and full encrypted snapshots are taken daily and retained for 35 days.

• ↳ Business Continuity and Disaster Recovery › Backups

Reference answer

Continuous backups with 7-day point-in-time recovery, plus daily encrypted full snapshots retained for 35 days, replicated to the DR location. Restores are tested monthly.

Expected: Business continuity › Backups

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies the continuous backup mechanism with 7-day point-in-time recovery and the daily encrypted full snapshots retained for 35 days, which aligns perfectly with the reference answer. However, the assistant omitted two key points from the reference: (1) the explicit mention of replication to the DR location, and (2) the monthly testing of restores. While the cited excerpts include these details, the assistant's answer did not explicitly state them. The assistant's claims are fully supported by the excerpts, but the omission of critical points reduces completeness. The main facts are correct, but the answer is not fully comprehensive as per the reference.

Drafted answer

Vantrel supports SAML 2.0 and OpenID Connect (OIDC) Single Sign-On (SSO) protocols, which are available on Business and Enterprise plans.

• ↳ Identity and Access Management › Customer authentication

Reference answer

SAML 2.0 and OpenID Connect, on Business and Enterprise plans. Tested with Microsoft Entra ID, Okta and Google Workspace. Enforcing SSO disables password login.

Expected: Identity access › Customer authentication

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies the supported SSO protocols (SAML 2.0 and OpenID Connect) and the plan restrictions (Business and Enterprise). However, it omits key details from the reference answer: the tested identity providers (Microsoft Entra ID, Okta, and Google Workspace) and the enforcement behavior (disabling password login when SSO is enforced). The cited excerpts fully support the assistant's claims, but the answer lacks these critical points, reducing completeness. The main facts are correct, but the omission of these details impacts the completeness score.

Drafted answer

Yes, user provisioning and deprovisioning through SCIM 2.0 is supported on the Enterprise plan.

• ↳ Identity and Access Management › User provisioning

Reference answer

Yes, SCIM 2.0 is supported, on the Enterprise plan only.

Expected: Identity access › User provisioning

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer aligns closely with the reference answer, correctly stating that SCIM 2.0 is supported and restricted to the Enterprise plan. The cited excerpt explicitly confirms this, ensuring both correctness and faithfulness. The assistant's answer is also complete, as it covers the key points of the reference (support for SCIM 2.0 and the Enterprise plan restriction) without omissions or unsupported additions.

Drafted answer

For Enterprise customers, we commit to a monthly uptime of 99.95%, excluding scheduled maintenance.

• ↳ Service Level Agreement and Support › Availability commitment

Reference answer

99.95% monthly uptime for Enterprise (excluding scheduled maintenance), with service credits of 10% or 25% of the monthly fee.

Expected: Sla support › Availability commitment

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer correctly states the **99.95% monthly uptime commitment for Enterprise customers**, which matches the reference answer. However, the assistant omitted the **service credits clause (10% or 25% of the monthly fee)**, which is a key detail in the reference. The cited excerpts fully support the uptime percentage and the exclusion of scheduled maintenance, but the service credits are missing from both the answer and the excerpts. The assistant did not hallucinate or add unsupported details, but the omission of a critical component reduces completeness.

Drafted answer

Vantrel notifies the customer of any personal data breach affecting the customer's data without undue delay, and in any case within 48 hours after becoming aware of it.

• ↳ Data Processing Agreement (summary) › Personal data breach notification
• ↳ Incident Response › Customer communication (not in expected sources)

Reference answer

Without undue delay and in any case within 48 hours after becoming aware of it.

Expected: Data processing agreement › Personal data breach notification

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer directly matches the reference answer, which states that notification of a personal data breach must occur 'without undue delay and in any case within 48 hours after becoming aware of it.' The assistant's response is fully supported by the cited excerpt from the **Data Processing Agreement (summary)**, which explicitly states the same timeframe. The assistant did not add any unsupported details or miss any key points, and the answer is concise and accurate. The additional excerpt about incident response (e.g., status updates, post-incident reports) is irrelevant to the question and does not affect the evaluation.

Drafted answer

Vantrel Plan is tested by an independent third-party security firm at least once a year, and additionally before major architectural changes. The most recent penetration test was completed in March 2026 and identified no critical findings and two high-severity findings, both of which were remediated within 30 days and verified by a retest.

• ↳ Penetration Testing and Vulnerability Management › Penetration testing

Reference answer

At least annually by an independent third party, plus before major architectural changes. The last test (March 2026) found no critical and two high findings, both remediated within 30 days and verified by retest.

Expected: Vulnerability management › Penetration testing

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is **fully consistent** with the reference answer in terms of frequency (annual + pre-major changes), timing (March 2026), and findings (no critical, two high-severity, remediated within 30 days with retest verification). The cited excerpts **directly support every claim** made, including the scope (web app, API, infrastructure) and the remediation process. The answer is **complete** as it covers all key points from the reference without omissions or unsupported additions. No contradictions or missing qualifications exist.

Drafted answer

Vantrel remediates critical vulnerabilities within 7 days.

• ↳ Penetration Testing and Vulnerability Management › Vulnerability management

Reference answer

Critical vulnerabilities must be remediated within 7 days (high: 30 days, medium: 90 days).

Expected: Vulnerability management › Vulnerability management

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer correctly states the remediation timeline for critical vulnerabilities as 7 days, which matches the reference answer and is directly supported by the cited excerpt. However, the assistant's answer omits the timelines for high and medium vulnerabilities (30 and 90 days, respectively), which are key points in the reference answer. This omission affects completeness but does not impact correctness or faithfulness, as the core claim about critical vulnerabilities is accurate and supported by the documentation. The assistant did not hallucinate or add unsupported details, and the cited excerpt fully backs the provided statement.

Drafted answer

Vantrel uses the following infrastructure sub-processors: Amazon Web Services EMEA SARL, Cloudflare, Inc., and Datadog, Inc. The service sub-processors are Stripe Payments Europe, Ltd., Zendesk, Inc., SendGrid (Twilio Inc.), and Microsoft Ireland Operations Ltd. (Azure OpenAI Service).

• ↳ Sub-processors › Infrastructure sub-processors
• ↳ Sub-processors › Service sub-processors

Reference answer

AWS (hosting), Cloudflare (CDN/WAF/DDoS), Datadog (monitoring), Stripe (billing), Zendesk (support), SendGrid/Twilio (transactional emails), Microsoft Azure OpenAI Service (AI features).

Expected: Subprocessors › Infrastructure sub-processors · Subprocessors › Service sub-processors

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer aligns perfectly with the reference answer in terms of the sub-processors listed (AWS, Cloudflare, Datadog, Stripe, Zendesk, SendGrid/Twilio, and Microsoft Azure OpenAI Service). The assistant's phrasing ('infrastructure' vs. 'service' sub-processors) is a minor organizational distinction but does not affect correctness. Every claim in the assistant's answer is directly supported by the cited excerpts, and no key points from the reference are omitted. The assistant's answer is comprehensive and accurate, with no hallucinations or unsupported additions.

Drafted answer

Customer data is never used to train or fine-tune AI models, neither by Vantrel nor by its AI sub-processor.

• ↳ AI Features Policy › Use of customer data

Reference answer

No. Customer data is never used to train or fine-tune AI models, neither by Vantrel nor by its AI sub-processor.

Expected: Ai features policy › Use of customer data

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is **fully consistent** with the reference answer, which states that customer data is never used to train or fine-tune AI models by Vantrel or its sub-processor. The cited excerpts directly support this claim, and no additional unsupported statements or omissions are present. The assistant's response is concise, accurate, and aligns perfectly with the reference and documentation.

Drafted answer

Yes, Single Sign-On (SSO) via OpenID Connect and SAML 2.0 is supported on Business and Enterprise plans, and Okta is one of the tested identity providers.

• ↳ Identity and Access Management › Customer authentication

Reference answer

Yes, via SAML 2.0 or OIDC SSO on Business and Enterprise plans; Okta is a tested identity provider.

Expected: Identity access › Customer authentication

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly states that SSO via OpenID Connect and SAML 2.0 is supported on Business and Enterprise plans, and that Okta is a tested identity provider, aligning perfectly with the reference answer. The cited excerpts explicitly mention these protocols and Okta as a tested identity provider, ensuring both faithfulness and correctness. The assistant's response is also fully complete, covering all key points from the reference without omissions or unsupported additions.

Drafted answer

In the event of a major incident affecting an entire AWS region such as Paris, customer data at rest stays within the selected region and is replicated to the disaster recovery location, with a defined Recovery Point Objective (RPO) of 1 hour.

• ↳ Infrastructure and Hosting › Hosting regions
• ↳ Business Continuity and Disaster Recovery › Backups (not in expected sources)
• ↳ Business Continuity and Disaster Recovery › Objectives

Reference answer

At most about 1 hour of data (RPO 1 hour), with service restored from the Frankfurt DR region within 4 hours (RTO).

Expected: Business continuity › Objectives · Infrastructure hosting › Hosting regions

Grader · correctness 3 · faithfulness 3 · completeness 3

The assistant's answer correctly identifies the RPO of 1 hour and mentions the replication of data to the disaster recovery location (Frankfurt), which aligns with the reference answer. However, it omits the critical RTO of 4 hours for service restoration, a key point in the reference. The assistant's answer also does not explicitly state the potential data loss duration (e.g., 'at most about 1 hour of data'), though this is implied by the RPO. The cited excerpts support the RPO and replication details but do not address the RTO or the explicit data loss duration. The assistant's answer is partially faithful (missing RTO) and incomplete (missing RTO and explicit data loss duration).

Drafted answer

Vantrel employees do not have standing access to customer data. Support engineers can only access a customer workspace when the customer grants temporary access from the administration console.

• ↳ Identity and Access Management › Vantrel employee access to customer data

Reference answer

No standing access. Support can only access a workspace when the customer grants temporary access (max 72 hours), which is logged and visible in the audit trail.

Expected: Identity access › Vantrel employee access to customer data

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant's answer correctly states that Vantrel employees lack standing access to customer data and that support engineers require temporary access granted by the customer. However, the assistant's answer omits the **time limit (72 hours)** and the **audit trail visibility** mentioned in both the reference and the cited excerpts, which are key details. While the core point (no standing access) is correct, the omission of these qualifications reduces the score for completeness. The assistant's claims are fully supported by the cited excerpts, so faithfulness is high. The main point aligns with the reference, but the missing details slightly lower correctness and completeness.

Drafted answer

Yes, Enterprise customers can provide and control their own encryption key through AWS KMS (Bring Your Own Key), which allows them to make their data unreadable by Vantrel if the key is revoked.

• ↳ Security Whitepaper › Encryption

Reference answer

Yes, Enterprise customers can use customer-managed keys (BYOK) via AWS KMS; revoking the key makes the data unreadable by Vantrel.

Expected: Security whitepaper › Encryption

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer aligns closely with the reference answer, correctly stating that Enterprise customers can use customer-managed keys (BYOK) via AWS KMS and that revoking the key renders the data unreadable by Vantrel. The cited excerpts directly support the claim about BYOK and its implications, including the revocation effect. The assistant's phrasing is consistent with the reference, and no key points are omitted. The only minor difference is the use of 'Bring Your Own Key' (BYOK) terminology, which is equivalent to 'customer-managed keys' and does not affect correctness or completeness. The answer is fully supported by the provided documentation excerpts, with no unsupported additions or omissions of key details.

Drafted answer

Background checks are performed on all new employees before their start date, where permitted by local law.

• ↳ Security Whitepaper › Employee security

Reference answer

Yes, background checks are performed on all new employees before their start date, where permitted by local law.

Expected: Security whitepaper › Employee security

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer directly matches the reference answer in both content and phrasing, confirming that background checks are performed on all new employees before their start date, where permitted by local law. The cited excerpt explicitly supports this claim without any deviations, additions, or omissions. The reference answer does not require further details beyond this core statement, and the assistant avoids introducing irrelevant or unsupported information (e.g., training, phishing simulations, or confidentiality agreements).

Drafted answer

Response times depend on your subscription plan. For Enterprise customers, the first-response target for P1 (service unavailable) is 1 hour, 24/7. For Business customers, the target is 4 hours during business hours (Monday–Fri). Starter customers have access to email support and the help center on a best-effort basis.

• ↳ Service Level Agreement and Support › Support

Reference answer

Enterprise: P1 first response within 1 hour, 24/7. Business: support is business hours only, P1 first response within 4 hours.

Expected: Sla support › Support

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer is **correct** as it aligns perfectly with the reference answer for Enterprise and Business plans, including the 24/7 availability for Enterprise and the 4-hour response time for Business during business hours. The assistant also **faithfully** cites the excerpts, as all claims (e.g., 1-hour P1 response for Enterprise, 4-hour P1 response for Business, and Starter's best-effort support) are directly supported by the table and accompanying text. However, the **completeness** is slightly reduced because the reference does not explicitly mention the Starter plan's details (best-effort support), so including this as a key point adds minor extra detail beyond the reference's scope. The core focus (Enterprise/Business) is fully covered, though.

Drafted answer

Yes. Sub-processors such as Stripe Payments Europe, Ltd. and SendGrid (Twilio Inc.) process data in the US, and Cloudflare, Inc. operates on a global edge network. Additionally, for customers with US-hosted tenants or depending on the selected services, infrastructure and monitoring data may be processed in the US.

• ↳ Sub-processors › Service sub-processors
• ↳ Sub-processors › Infrastructure sub-processors

Reference answer

For EU tenants, hosting (AWS), monitoring (Datadog), support (Zendesk) and AI (Azure EU Data Zone) stay in the EU. Cloudflare handles traffic in transit on its global network, Stripe operates in EU/US, and SendGrid processes email addresses and notification content in the US. Transfers are covered by SCCs; Vantrel Inc. is under the EU-U.S. Data Privacy Framework.

Expected: Subprocessors › Infrastructure sub-processors · Subprocessors › Service sub-processors · Data processing agreement › International data transfers

Grader · correctness 3 · faithfulness 3 · completeness 3

The assistant's answer correctly identifies that Stripe, SendGrid, and Cloudflare process data outside the EU, aligning with the reference answer. However, the assistant's mention of 'US-hosted tenants' and 'infrastructure and monitoring data processed in the US' introduces an unsupported generalization, as the reference specifies these only apply to non-EU tenants or specific service selections. The assistant also omits the reference's mention of SCCs and the EU-U.S. Data Privacy Framework, which are key to the completeness of the answer. The cited excerpts support the main claims but lack the broader context of data transfer mechanisms and tenant-specific restrictions, leading to a minor loss in faithfulness and completeness.

Drafted answer

Customers affected by a security incident involving their data are notified by email by the Security team, adhering to the timelines defined in the Data Processing Agreement, which specifies notification of a personal data breach without undue delay and within 48 hours after becoming aware of it. Additionally, service availability incidents are published on the public status page at status.vantrel.example.

• ↳ Incident Response › Customer communication
• ↳ Data Processing Agreement (summary) › Personal data breach notification

Reference answer

Availability incidents on the status page (updates at least every 60 min for SEV1); customers whose data is affected are emailed by Security within the DPA timeline (48 hours); a root cause analysis is shared within 10 business days after a SEV1.

Expected: Incident response › Customer communication · Data processing agreement › Personal data breach notification

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant's answer aligns closely with the reference answer in terms of correctness, covering the email notification for affected customers within 48 hours (as per the DPA) and the status page updates for availability incidents. However, the assistant omitted the key detail about the **root cause analysis being shared within 10 business days after a SEV1 incident**, which is explicitly mentioned in the reference and excerpts. The assistant’s claims are fully supported by the cited excerpts, but the omission of this critical point reduces completeness. The answer is otherwise faithful and correct, with no unsupported or incorrect statements.

Drafted answer

For customer users, multi-factor authentication (MFA) can be enabled using a TOTP authenticator app or a FIDO2/WebAuthn security key, and administrators can make MFA mandatory for all users of their workspace (SMS-based MFA is not supported). The documentation does not contain information regarding MFA options for Vantrel staff accessing production.

• ↳ Identity and Access Management › Customer authentication

Reference answer

Customers: TOTP or FIDO2/WebAuthn, can be made mandatory by admins, no SMS. Staff: production admin access goes through a bastion with SSO, MFA and session recording; direct SSH disabled.

Expected: Identity access › Customer authentication · Infrastructure hosting › Network security

Grader · correctness 3 · faithfulness 5 · completeness 1

The assistant's answer correctly identifies the MFA options for customer users (TOTP and FIDO2/WebAuthn) and the ability for admins to enforce MFA, aligning perfectly with the reference and cited documentation. However, the assistant incorrectly states that the documentation does not contain information regarding MFA options for Vantrel staff accessing production, when the reference explicitly states that staff production access requires a bastion with SSO, MFA, and session recording, with direct SSH disabled. This omission of key details for staff access results in a lower score for completeness. The assistant’s claims are fully supported by the cited excerpts for customer users, so faithfulness is high. The main point for customers is correct, but the staff section is entirely missing, which is critical for completeness.

Drafted answer

Infrastructure and application logs are centralized in Datadog, utilizing the EU site for EU-hosted tenants, and are retained for 12 months. Audit logs are retained for 13 months, but the documentation does not specify where audit logs are stored. Additionally, there is a conflict in the documentation regarding data deletion timelines after contract termination (30 days versus 90 days), though this specifically refers to customer data rather than logs.

• ↳ Infrastructure and Hosting › Logging and monitoring
• ↳ Identity and Access Management › Audit logs
• ↳ Security Whitepaper › Data retention and deletion (not in expected sources)
• ↳ Sub-processors › Infrastructure sub-processors (not in expected sources)
• ↳ Data Processing Agreement (summary) › Return and deletion of data (not in expected sources)

Reference answer

Infrastructure and application logs are centralized in Datadog (EU site for EU tenants), retained 12 months. Customer audit logs are retained 13 months and exportable via API or streamed to a SIEM (Enterprise).

Expected: Infrastructure hosting › Logging and monitoring · Identity access › Audit logs

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies the storage location (Datadog, EU site for EU tenants) and retention periods (12 months for infrastructure/application logs, 13 months for audit logs) as per the reference. However, it omits the key detail that audit logs are *also* stored in Datadog (implied by the reference's mention of exportability/API access, which aligns with Datadog's role). The assistant flags conflicting sources for data deletion timelines (30 vs. 90 days), but this is unrelated to logs and thus irrelevant to the question. The cited excerpts fully support the assistant's claims about logs, but the omission of audit log storage location (Datadog) reduces completeness. The assistant's mention of conflicting sources is accurate but tangential to the question, and the answer does not hallucinate or extrapolate unsupported claims beyond the excerpts provided for logs.

Drafted answer

Yes, when Vantrel Assist is enabled, customer data required to answer a specific request is sent to Microsoft Ireland Operations Ltd. (Azure OpenAI Service), which serves as the AI sub-processor. For EU-hosted tenants, this service is deployed in the EU Data Zone.

• ↳ AI Features Policy › Use of customer data
• ↳ Sub-processors › Service sub-processors
• ↳ AI Features Policy › Vantrel Assist
• ↳ Sub-processors › Notes (not in expected sources)

Reference answer

Only if a workspace Admin enables Vantrel Assist (off by default). Then only the data needed for the request is sent to Microsoft Azure OpenAI Service in the EU Data Zone; it is never used for training.

Expected: Ai features policy › Vantrel Assist · Ai features policy › Use of customer data · Subprocessors › Service sub-processors

Grader · correctness 4 · faithfulness 4 · completeness 4

The assistant's answer correctly identifies that data is sent to a third party (Microsoft Ireland Operations Ltd., Azure OpenAI Service) only when Vantrel Assist is enabled, which aligns with the reference answer. The assistant also accurately states that only the data required for the request is sent and that the service is deployed in the EU Data Zone for EU-hosted tenants. However, the assistant's answer does not explicitly mention that the data is *never used for training*, which is a key point in the reference answer and excerpts. The assistant's claim about Microsoft Ireland Operations Ltd. being the AI sub-processor is correct, but the reference answer specifies *Microsoft Azure OpenAI Service* as the third party, which is more precise. The assistant's answer is mostly faithful to the excerpts, but the omission of the 'never used for training' detail slightly reduces the faithfulness score. Completeness is slightly reduced due to the missing training detail, but the core points are covered.

Drafted answer

Vantrel runs a private bug bounty program with invited researchers as stated in the documentation. However, the documentation does not contain any information regarding reward amounts.

• ↳ Incident Response › Reporting a vulnerability

Reference answer

Yes, a private bug bounty program with invited researchers. Reward amounts are not documented.

Expected: Incident response › Reporting a vulnerability

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly identifies that Vantrel has a private bug bounty program with invited researchers, which aligns perfectly with the reference answer and the cited documentation. The assistant also accurately states that reward amounts are not documented, matching the reference answer. The cited excerpt directly supports the claim about the bug bounty program, and no unsupported claims or hallucinations are present. The answer is concise and covers all key points from the reference without omissions or additions beyond the scope of the documentation or reference.

Drafted answer

The documentation states that Vantrel is tested by an independent third-party security firm at least once a year, but the specific name or identity of the firm is not documented.

• ↳ Penetration Testing and Vulnerability Management › Penetration testing

Reference answer

An independent third-party security firm, in March 2026. The firm's name is not documented.

Expected: Vulnerability management › Penetration testing

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies that the last penetration test was performed by an independent third-party security firm (consistent with the reference) and that the firm's name is not documented (also consistent with the reference). However, the assistant's answer does not explicitly mention the date of the last penetration test (March 2026), which is a key detail in the reference answer. The assistant's answer is faithful to the cited excerpts, as it only states what is explicitly mentioned (the firm's identity is not documented, and testing is done by an independent third-party). The assistant's answer is mostly complete but misses the date of the last test, which is a key point in the reference. The assistant also does not address the question directly by stating the name of the firm, which the reference explicitly states is not documented (and thus cannot be answered).

Drafted answer

Enterprise customers can stream audit logs to their SIEM via webhook.

• ↳ Identity and Access Management › Audit logs

Reference answer

Enterprise customers can stream audit logs to their SIEM via webhook; specific supported SIEM products are not listed.

Expected: Identity access › Audit logs

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly states that enterprise customers can stream audit logs to their SIEM via webhook, which aligns perfectly with the reference answer. The reference explicitly mentions that no specific SIEM products are listed, and the assistant avoids making unsupported claims about supported products. The cited excerpt directly supports the assistant's statement, and the answer covers the key point of the reference without omissions or additions. Since the reference confirms the information is not documented beyond the webhook capability, the assistant's response is accurate and complete.

Drafted answer

The Data Protection Officer (DPO) can be contacted at dpo@vantrel.example.

• ↳ Company Overview › Security and compliance organization

Reference answer

The DPO can be contacted at dpo@vantrel.example; the DPO's name is not documented.

Expected: Company overview › Security and compliance organization

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies the contact email for the Data Protection Officer (DPO) as `dpo@vantrel.example`, which is explicitly stated in the cited excerpt. However, the reference answer clarifies that the DPO's name is **not documented**, a key point the assistant omitted. While the assistant's answer is factually correct for the provided email, it fails to address the completeness of the reference answer by not mentioning the absence of the DPO's name in the documentation. The cited excerpt fully supports the assistant's claim about the email, so faithfulness is not an issue here. The omission of the 'name not documented' detail reduces the completeness score.

Drafted answer

No, custom roles are not currently supported by Vantrel Plan.

• ↳ Identity and Access Management › Roles and permissions

Reference answer

No, custom roles are not currently supported. Four default roles exist (Admin, Modeler, Contributor, Viewer) plus dimension-member-level restrictions.

Expected: Identity access › Roles and permissions

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer correctly states that custom roles are not supported, aligning perfectly with the reference answer. However, the assistant's answer includes a qualification ('by Vantrel Plan') that is not explicitly stated in the cited excerpts or the reference answer, though it is implied. The assistant's response is faithful to the excerpts and covers the key point of the reference answer (no custom roles), but it misses the mention of the four default roles and dimension-member-level restrictions, which are key details in the reference. The assistant's answer is concise but lacks completeness in this regard.

Drafted answer

Vantrel does not offer on-premise, private cloud or self-hosted deployments. Vantrel Plan is delivered exclusively as a multi-tenant Software-as-a-Service (SaaS) application accessed through a web browser.

• ↳ Company Overview › Product and delivery model

Reference answer

No. Vantrel Plan is SaaS only; no on-premise, private cloud or self-hosted deployment.

Expected: Company overview › Product and delivery model

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is factually consistent with the reference answer, accurately stating that Vantrel Plan is SaaS-only with no on-premise, private cloud, or self-hosted options. The cited excerpt directly supports all claims, and the answer covers all key points from the reference without omissions or unsupported additions. The phrasing is slightly more detailed (e.g., 'exclusively multi-tenant SaaS') but does not introduce errors or misrepresentations.

Drafted answer

SMS-based MFA is not supported.

• ↳ Identity and Access Management › Customer authentication

Reference answer

No, SMS-based MFA is not supported (TOTP and FIDO2/WebAuthn are).

Expected: Identity access › Customer authentication

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer ('SMS-based MFA is not supported') is factually correct and aligns perfectly with the reference answer, which explicitly states that SMS-based MFA is not supported while listing TOTP and FIDO2/WebAuthn as supported alternatives. The assistant's answer is directly supported by the cited excerpt, which explicitly mentions 'SMS-based MFA is not supported' under the MFA section. The answer is also complete, as it covers the key point of the reference without omissions or unnecessary details. No contradictions or unsupported claims are present.

Drafted answer

No, data cannot be hosted in Australia. The available hosting regions are EU (Paris, France with disaster recovery in Frankfurt, Germany) and US (N. Virginia with disaster recovery in Oregon), and no other hosting regions are currently offered.

• ↳ Infrastructure and Hosting › Hosting regions

Reference answer

No. Only EU (Paris/Frankfurt) and US (N. Virginia/Oregon) regions are offered.

Expected: Infrastructure hosting › Hosting regions

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly states that data cannot be hosted in Australia, aligning perfectly with the reference answer. The assistant's answer also accurately lists the available hosting regions (EU and US) and their specific locations, including disaster recovery details, which are fully supported by the cited excerpts. The assistant's response is complete, covering all key points in the reference answer without omissions or unsupported claims. The only minor difference is the phrasing of 'no other hosting regions are currently offered,' which is consistent with the reference and excerpts but slightly more explicit. This does not affect correctness or faithfulness.

Drafted answer

An accessibility conformance report (VPAT) is not yet available, although Vantrel is working toward conformance with WCAG 2.1 level AA.

• ↳ Product FAQ › Is Vantrel Plan accessible to users with disabilities?

Reference answer

Not yet. Vantrel is working toward WCAG 2.1 AA, but a VPAT is not available yet.

Expected: Product faq › Is Vantrel Plan accessible to users with disabilities?

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer aligns perfectly with the reference answer, accurately stating that a VPAT is not yet available and that Vantrel is working toward WCAG 2.1 AA. The cited excerpt directly supports both claims, ensuring faithfulness. The answer is also complete, covering all key points from the reference without omissions or unsupported additions.

Drafted answer

The provided documentation does not contain information regarding FedRAMP authorization for Vantrel Plan.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 3 · completeness 5

The assistant correctly identifies that the documentation does not contain information about FedRAMP authorization for Vantrel Plan, which aligns perfectly with the reference answer. However, the assistant did not cite any excerpts to support this claim, which is unusual since the reference explicitly states the information is *not* documented. Since the reference confirms the absence of documentation, the assistant’s answer is factually correct, but the lack of cited excerpts prevents a full score for faithfulness. Completeness is fully satisfied as the answer directly addresses the question without omissions or additions.

Drafted answer

The provided documentation does not contain information on whether Vantrel will sign a HIPAA Business Associate Agreement (BAA).

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the documentation does not address the question of signing a HIPAA BAA, aligning perfectly with the reference answer. However, the assistant did not cite any excerpts to support this conclusion, which is a minor oversight since the reference answer itself states the information is not documented. The reasoning is sound, and the answer is complete as it directly mirrors the reference. Since no excerpts were cited, faithfulness cannot be fully evaluated, but the absence of unsupported claims means no deduction is necessary for this criterion.

Drafted answer

The provided documentation does not contain information about the coverage amount of our cyber insurance policy.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the coverage amount of the cyber insurance policy is not documented, which aligns perfectly with the reference answer. However, the assistant did not cite any excerpts to support this claim, which is unusual since the reference explicitly states the information is absent. Despite this, the absence of documentation is a factual statement that does not require citation, so faithfulness is not penalized. The answer is concise and covers the key point fully without unnecessary details or inaccuracies.

Drafted answer

The provided documentation does not contain information regarding source code escrow agreements.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 3 · completeness 5

The assistant correctly identifies that the documentation does not address the question of a source code escrow agreement, aligning perfectly with the reference answer. However, the assistant did not cite any excerpts, which is a minor procedural issue but does not affect the correctness or completeness of the response. Since the reference explicitly states the information is not documented, the answer is fully accurate and complete, though the lack of citation prevents a perfect faithfulness score (as no excerpts were provided to verify).

Drafted answer

The provided documentation does not contain information regarding IP allowlisting to restrict user access.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the documentation does not address IP allowlisting for restricting user access, which aligns perfectly with the reference answer. Since the reference explicitly states the information is not documented, the assistant’s response is accurate and complete. However, the assistant did not cite any excerpts, which is acceptable here because the reference confirms the absence of documentation, but it would have been better to explicitly state that no relevant excerpts were found. The answer is factually correct, faithful to the reference, and fully complete.

Drafted answer

The provided documentation does not contain information regarding the use of post-quantum cryptography.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the documentation does not address the use of post-quantum cryptography, aligning perfectly with the reference answer. However, the assistant did not cite any excerpts, which is a minor procedural issue but does not affect the correctness or completeness of the response. Since the reference explicitly states the information is not documented, the answer is fully accurate and complete, though the lack of cited excerpts slightly impacts faithfulness (as no excerpts were provided to verify).

Drafted answer

Vantrel provides a SOC 2 Type II report and an ISO 27001 certificate rather than a SOC 1 report.

• ↳ Data Processing Agreement (summary) › Audits
• ↳ Security Whitepaper › Certifications and audit reports

Reference answer

Not in the documentation. Only Vantrel's SOC 2 Type II is documented; the SOC 1 mentioned is AWS's, not Vantrel's.

Grader · correctness 5 · faithfulness 5 · completeness 4

The assistant's answer correctly identifies that Vantrel does not provide a SOC 1 report, aligning with the reference answer. However, the assistant's answer does not explicitly state that the SOC 1 report mentioned is AWS's, not Vantrel's, which is a key distinction in the reference. The assistant's claims about SOC 2 Type II and ISO 27001 are fully supported by the cited excerpts, and the answer covers the key points of the reference (absence of SOC 1, presence of SOC 2 and ISO 27001). The assistant did not claim the information was 'not documented' but instead provided accurate details about what *is* documented, which is consistent with the reference's implication that SOC 1 is not provided by Vantrel. The assistant's answer is factually correct, well-supported, and complete for the key points, though it misses the AWS clarification from the reference.

Drafted answer

The provided documentation does not contain information on the number of people who work in the security team.

Reference answer

Not in the documentation. Only the total headcount (~240) and the CISO/Security Committee are mentioned.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the documentation does not provide the specific number of people in the security team, aligning perfectly with the reference answer. However, the assistant did not cite any excerpts, which is a minor oversight since the reference answer itself is based on the absence of information rather than specific citations. The answer is concise, accurate, and covers the key point of the reference (lack of documentation).

Drafted answer

The documentation does not provide a mean time to resolve (MTTR) for P1 incidents. The response times for P1 incidents are first-response targets of 1 hour for Enterprise and 4 hours for Business customers, which are not resolution times.

• ↳ Service Level Agreement and Support › Support

Reference answer

Not in the documentation. Only first-response targets are documented, explicitly not resolution times.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the documentation does **not** provide the mean time to resolve (MTTR) for P1 incidents, aligning perfectly with the reference answer. The assistant also accurately cites the first-response targets for P1 incidents (1 hour for Enterprise, 4 hours for Business) and clarifies that these are **not** resolution times, as explicitly stated in the excerpts. The answer is fully faithful to the documentation and covers all key points of the reference (lack of MTTR documentation and the distinction between first-response and resolution times).

Drafted answer

The provided documentation does not contain information regarding CSA STAR certification.

Reference answer

Not in the documentation. The documented certifications are ISO 27001 and SOC 2 Type II.

Grader · correctness 5 · faithfulness 3 · completeness 5

The assistant correctly identifies that the information about CSA STAR certification is not documented, which aligns perfectly with the reference answer. However, the assistant did not cite any excerpts to support this claim, which is a minor but notable omission given the requirement to reference documentation. The answer is concise and avoids incorrect assumptions or unsupported claims, focusing solely on the absence of information in the provided documentation. Since the reference explicitly states the information is not documented, the assistant’s response is fully correct and complete, though the lack of cited excerpts slightly impacts the faithfulness score.

Drafted answer

The provided documentation does not contain information about an emergency phone number for security incidents.

Reference answer

Not in the documentation. Only email (security@vantrel.example) and the status page are documented.

Grader · correctness 5 · faithfulness 3 · completeness 5

The assistant correctly identifies that the documentation does not provide an emergency phone number for security incidents, which aligns perfectly with the reference answer. However, the assistant did not cite any excerpts to support this claim, which is unusual since the reference explicitly mentions documented alternatives (email and status page). The absence of citations means the answer lacks direct evidence, though the conclusion itself is accurate. Since the reference confirms the information is not documented, the correctness and completeness scores are maximized, but faithfulness is slightly impacted due to the lack of cited evidence.

Drafted answer

There is a contradiction in the documentation regarding the deletion timeframe after contract termination. According to the Security Whitepaper (v2026.1), customer data is permanently deleted from production systems within 90 days of contract termination (S1). However, the Data Processing Agreement summary (v2025.3) states that data is deleted within 30 days after the end of the 30-day grace period (S2).

• ↳ Security Whitepaper › Data retention and deletion
• ↳ Data Processing Agreement (summary) › Return and deletion of data

Reference answer

The documentation is inconsistent: the Security Whitepaper says within 90 days of termination, while the DPA says within 30 days after a 30-day export grace period (i.e. ~60 days). Both sources should be cited and the inconsistency flagged.

Expected: Security whitepaper › Data retention and deletion · Data processing agreement › Return and deletion of data

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identified the contradiction between the two sources (Security Whitepaper and DPA) regarding the deletion timeframe after contract termination. The assistant accurately cited both excerpts and reproduced their key details (90 days in the Whitepaper vs. ~60 days in the DPA, accounting for the 30-day grace period). The reference answer explicitly required flagging the inconsistency, which the assistant did, and the assistant’s answer aligns perfectly with the reference’s expectations. No facts are misstated or omitted beyond the documented contradiction, and all claims are directly supported by the cited excerpts. The assistant’s answer is thus fully correct, faithful, and complete for the given question and sources.

Drafted answer

Pour les clients de la région EU (par défaut), les données sont hébergées sur AWS eu-west-3 (Paris, France), avec un emplacement de reprise après sinistre à Francfort, en Allemagne. Le stockage des données au repos, y compris les sauvegardes, reste dans la région sélectionnée.

• ↳ Infrastructure and Hosting › Hosting regions

Reference answer

Oui pour la région EU (par défaut) : AWS eu-west-3 à Paris, avec reprise d'activité à Francfort (Allemagne). Une région US existe aussi.

Expected: Infrastructure hosting › Hosting regions

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer correctly states that EU-region data is hosted in Paris (AWS eu-west-3) with disaster recovery in Frankfurt, aligning perfectly with the reference. It also accurately mentions that data at rest (including backups) remains within the selected region, which is a key detail in the reference. However, the assistant's answer does not explicitly mention the existence of a US region (only implied by 'Le stockage des données au repos... reste dans la région sélectionnée'), which is a key point in the reference. The cited excerpts fully support all claims made, and the answer is mostly complete but misses the explicit mention of the US region as a secondary hosting option. The assistant's reasoning is faithful to the documentation and avoids hallucination or unsupported claims.

Drafted answer

Oui, Vantrel prend en charge l'authentification unique (SSO) avec Microsoft Entra ID via SAML 2.0 et OpenID Connect (OIDC) sur les plans Business et Enterprise.

• ↳ Identity and Access Management › Customer authentication

Reference answer

Oui, SSO SAML 2.0 et OIDC sur les plans Business et Enterprise ; Microsoft Entra ID fait partie des fournisseurs testés.

Expected: Identity access › Customer authentication

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly states that Vantrel supports SSO with Microsoft Entra ID via SAML 2.0 and OIDC on Business and Enterprise plans, which aligns perfectly with the reference answer. The cited excerpt explicitly confirms this, including the mention of Microsoft Entra ID as a tested provider. The assistant’s answer is fully supported by the documentation and covers all key points without omissions or unsupported claims. The only minor difference is the phrasing ('Vantrel takes in charge' vs. 'supported'), which does not affect correctness or completeness.

Drafted answer

The provided documentation does not contain information regarding SecNumCloud qualification by ANSSI.

Reference answer

Pas dans la documentation.

Grader · correctness 5 · faithfulness 3 · completeness 5

The assistant correctly identifies that the documentation does not address the question of SecNumCloud qualification by ANSSI, aligning perfectly with the reference answer. However, the assistant did not cite any excerpts, which is a minor procedural issue but does not affect the factual accuracy or completeness of the response. Since the reference explicitly states the information is not documented, the answer is both correct and complete, though the lack of cited excerpts slightly undermines faithfulness in a procedural sense (as no excerpts were provided to verify).