Drafted by Gemini, graded by Mistral

Drafted answer

All customer data stored in databases, file storage, and backups is encrypted at rest using AES-256. Keys are managed in AWS Key Management Service and are rotated automatically every year.

• ↳ Security Whitepaper › Encryption

Reference answer

Yes. All customer data (databases, file storage, backups) is encrypted at rest with AES-256, keys managed in AWS KMS and rotated automatically every year.

Expected: Security whitepaper › Encryption

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is **fully consistent** with the reference answer in terms of correctness, as it accurately states that customer data (databases, file storage, backups) is encrypted at rest using AES-256, with keys managed in AWS KMS and rotated annually. The cited excerpts **directly support** all claims, and the answer **covers all key points** without omissions or unsupported additions. The reference does not mention any contradictions or missing qualifications, and the assistant did not introduce any incorrect or extraneous details. The only minor deviation is the inclusion of the 'Customer-managed keys (BYOK)' excerpt, which is irrelevant to the question but does not detract from the score since it does not affect correctness, faithfulness, or completeness for the asked question.

Drafted answer

All connections to Vantrel Plan use TLS 1.2 or higher, with TLS 1.3 preferred and negotiated when supported by the client, while older protocols such as SSL, TLS 1.0, and 1.1 are disabled.

• ↳ Security Whitepaper › Encryption

Reference answer

TLS 1.2 is the minimum; TLS 1.3 is preferred. SSL, TLS 1.0 and 1.1 are disabled, and HSTS is enforced.

Expected: Security whitepaper › Encryption

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer aligns perfectly with the reference answer in terms of correctness, as it accurately states that TLS 1.2 is the minimum supported version, TLS 1.3 is preferred, and older protocols (SSL, TLS 1.0, and 1.1) are disabled. The assistant also correctly includes the enforcement of HSTS, which is a key point in the reference. The cited excerpts fully support all claims made in the answer, ensuring faithfulness. The answer covers all key points from the reference without omissions or unsupported additions, demonstrating completeness.

Drafted answer

Yes, Vantrel has been certified to ISO/IEC 27001:2022 since September 2023. The certificate covers the development, operation and support of Vantrel Plan.

• ↳ Security Whitepaper › Certifications and audit reports

Reference answer

Yes, ISO/IEC 27001:2022 certified since September 2023, covering development, operation and support of Vantrel Plan, with annual surveillance audits.

Expected: Security whitepaper › Certifications and audit reports

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is **correct** as it fully aligns with the reference answer, including the certification date (September 2023), the ISO/IEC 27001:2022 standard, and the scope (development, operation, and support of Vantrel Plan). The **faithfulness** is perfect since every claim is directly supported by the cited excerpt from the Security Whitepaper. The **completeness** is also excellent, as the assistant covers all key points from the reference (certification status, date, and scope) without omissions or irrelevant additions. The SOC 2 excerpt, while cited, is not relevant to the question and does not affect the scores.

Drafted answer

Yes, we undergo an annual SOC 2 Type II examination covering the Security, Availability, and Confidentiality Trust Services Criteria, and the report can be shared with customers and prospects under a non-disclosure agreement upon request to security@vantrel.example.

• ↳ Data Processing Agreement (summary) › Audits (not in expected sources)
• ↳ Security Whitepaper › Certifications and audit reports

Reference answer

Yes. Vantrel has an annual SOC 2 Type II (Security, Availability, Confidentiality; period Jan 1 - Dec 31, report available around March). It is shared under NDA on request to security@vantrel.example.

Expected: Security whitepaper › Certifications and audit reports

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is factually consistent with the reference answer, correctly stating that Vantrel provides an annual SOC 2 Type II report covering Security, Availability, and Confidentiality, and that it can be shared under NDA upon request. The cited excerpts fully support these claims, including the audit period (Jan 1 - Dec 31) and the sharing condition. The assistant's answer is also complete, covering all key points from the reference (availability, scope, and sharing mechanism). No contradictions or omissions are present, and the answer aligns with the documentation excerpts provided. The assistant did not introduce unsupported details or extrapolate beyond the cited sources.

Drafted answer

For customers hosted in the EU region, customer data is stored and processed in the European Union, with the primary location in Paris, France and the disaster recovery location in Frankfurt, Germany. Customer data at rest stays within the selected region, including backups and disaster recovery copies.

• ↳ Data Processing Agreement (summary) › International data transfers (not in expected sources)
• ↳ Infrastructure and Hosting › Hosting regions

Reference answer

On AWS in eu-west-3 (Paris, France), with disaster recovery in eu-central-1 (Frankfurt, Germany). Data at rest, including backups, stays in the selected region.

Expected: Infrastructure hosting › Hosting regions

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer aligns closely with the reference answer, correctly identifying the primary hosting location (AWS eu-west-3, Paris) and disaster recovery location (AWS eu-central-1, Frankfurt) for EU customers. It also accurately states that data at rest, including backups, remains within the selected region. The assistant's phrasing is consistent with the excerpts, though it omits the minor detail about limited remote access by support staff (which does not affect correctness or completeness). The cited excerpts fully support the claims made, and all key points from the reference are covered without omissions or errors. The answer is precise and avoids unsupported extrapolations or hallucinations.

Drafted answer

For Vantrel Plan, our Recovery Point Objective is 1 hour and our Recovery Time Objective is 4 hours in the event of a major incident affecting an entire AWS region.

• ↳ Business Continuity and Disaster Recovery › Objectives

Reference answer

RPO of 1 hour and RTO of 4 hours.

Expected: Business continuity › Objectives

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly aligns with the reference answer by stating the RPO of 1 hour and RTO of 4 hours for the Vantrel Plan. The cited excerpts directly support these values, and the assistant's phrasing ('in the event of a major incident affecting an entire AWS region') is consistent with the context provided in the documentation. The answer is precise, avoids unsupported additions, and covers all key points from the reference without omissions or inaccuracies.

Drafted answer

Databases are backed up continuously with point-in-time recovery available for the last 7 days, and full encrypted snapshots are taken daily and retained for 35 days. Backups containing customer data expire according to the backup retention schedule after contract termination.

• ↳ Business Continuity and Disaster Recovery › Backups
• ↳ Security Whitepaper › Data retention and deletion (not in expected sources)

Reference answer

Continuous backups with 7-day point-in-time recovery, plus daily encrypted full snapshots retained for 35 days, replicated to the DR location. Restores are tested monthly.

Expected: Business continuity › Backups

Grader · correctness 3 · faithfulness 3 · completeness 3

The assistant's answer correctly captures the core backup frequency (continuous with 7-day point-in-time recovery and daily encrypted snapshots retained for 35 days) and aligns with the reference answer. However, the assistant's inclusion of 'Backups containing customer data expire according to the backup retention schedule after contract termination' is misleadingly vague-it omits the critical detail from the *Security Whitepaper* that backups expire **within 90 days** of contract termination (not tied to the 35-day retention window for active subscriptions). The reference also explicitly mentions DR replication and monthly restore testing, which the assistant omits entirely. The cited excerpts fully support the assistant’s claims except for the unsupported extrapolation about post-contract expiration timing. The reference’s key points (DR replication, restore testing, and 90-day post-contract deletion) are missing, reducing completeness. Faithfulness is high but not perfect due to the unsupported claim about expiration timing.

Drafted answer

We support SAML 2.0 and OpenID Connect (OIDC) protocols for Single Sign-On, which are available on Business and Enterprise plans.

• ↳ Identity and Access Management › Customer authentication

Reference answer

SAML 2.0 and OpenID Connect, on Business and Enterprise plans. Tested with Microsoft Entra ID, Okta and Google Workspace. Enforcing SSO disables password login.

Expected: Identity access › Customer authentication

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies the supported SSO protocols (SAML 2.0 and OpenID Connect) and the plan restrictions (Business and Enterprise). However, it omits key details from the reference answer: the tested identity providers (Microsoft Entra ID, Okta, and Google Workspace) and the enforcement behavior (disabling password login when SSO is enforced). The cited excerpts fully support the assistant's claims, but the answer lacks these critical points, resulting in a completeness score of 3. The assistant did not hallucinate or extrapolate unsupported information, so faithfulness is high (5). The main facts are correct, but the omission of key details reduces the correctness score to 3 due to missing qualifications (e.g., enforcement impact).

Drafted answer

We support user provisioning and deprovisioning through SCIM 2.0 on the Enterprise plan.

• ↳ Identity and Access Management › User provisioning

Reference answer

Yes, SCIM 2.0 is supported, on the Enterprise plan only.

Expected: Identity access › User provisioning

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer aligns perfectly with the reference answer in terms of correctness: it correctly states that SCIM 2.0 is supported and restricted to the Enterprise plan. The cited excerpt directly supports the assistant's claim, ensuring faithfulness. The assistant's answer also covers all key points from the reference-support for SCIM 2.0 and the Enterprise plan restriction-without omissions or unnecessary additions, fulfilling completeness. The reasoning and citations are precise and accurate.

Drafted answer

For Enterprise customers, we commit to a monthly uptime of 99.95%, excluding scheduled maintenance.

• ↳ Service Level Agreement and Support › Availability commitment

Reference answer

99.95% monthly uptime for Enterprise (excluding scheduled maintenance), with service credits of 10% or 25% of the monthly fee.

Expected: Sla support › Availability commitment

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer correctly states the **99.95% monthly uptime commitment for Enterprise customers**, which matches the reference answer. However, the reference also explicitly mentions **service credits of 10% or 25% of the monthly fee** in case of downtime, which the assistant omitted. The cited excerpts support the uptime percentage but do not mention service credits, so the assistant's answer is **partially incomplete** and **lacks a key detail** from the reference. The assistant did not hallucinate or misrepresent facts, but the omission of service credits reduces the completeness score.

Drafted answer

We notify the customer of any personal data breach affecting their data without undue delay, and in any case within 48 hours after becoming aware of it.

• ↳ Data Processing Agreement (summary) › Personal data breach notification

Reference answer

Without undue delay and in any case within 48 hours after becoming aware of it.

Expected: Data processing agreement › Personal data breach notification

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is factually consistent with the reference answer, as both state the notification must occur 'without undue delay' and 'within 48 hours' after becoming aware of the breach. The cited excerpt from the Data Processing Agreement directly supports the assistant's response, including the key timeframe and the obligation to notify. The answer is concise, covering all key points without unnecessary details or deviations.

Drafted answer

Vantrel is tested by an independent third-party security firm at least once a year, and additionally before major architectural changes. The most recent penetration test was completed in March 2026 and identified no critical findings and two high-severity findings, both of which were remediated within 30 days and verified by the testing firm during a retest.

• ↳ Penetration Testing and Vulnerability Management › Penetration testing

Reference answer

At least annually by an independent third party, plus before major architectural changes. The last test (March 2026) found no critical and two high findings, both remediated within 30 days and verified by retest.

Expected: Vulnerability management › Penetration testing

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is **fully consistent** with the reference answer in terms of frequency (annual + pre-major architectural changes), timing (March 2026), and findings (no critical, two high-severity, remediated within 30 days). The cited excerpts **directly support** every claim, including the remediation timeline and verification process. The answer is also **complete**, covering all key points from the reference without omissions or unsupported additions. No contradictions or missing qualifications exist.

Drafted answer

Vantrel remediates critical vulnerabilities within 7 days.

• ↳ Penetration Testing and Vulnerability Management › Vulnerability management

Reference answer

Critical vulnerabilities must be remediated within 7 days (high: 30 days, medium: 90 days).

Expected: Vulnerability management › Vulnerability management

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly states the remediation timeline for critical vulnerabilities as 7 days, which aligns perfectly with the reference answer and the cited documentation. The assistant did not include the timelines for high and medium vulnerabilities, but since the question specifically asks for critical vulnerabilities, this omission does not affect correctness or faithfulness. The answer is concise, supported by the cited excerpt, and fully covers the key point for critical vulnerabilities, making it complete for the scope of the question.

Drafted answer

Our infrastructure and service sub-processors include Amazon Web Services EMEA SARL for cloud hosting, storage, and backups, Cloudflare, Inc. for CDN, WAF, and DDoS protection, Datadog, Inc. for infrastructure and application monitoring and logs, Stripe Payments Europe, Ltd. for billing and subscription payments, Zendesk, Inc. for customer support ticketing, SendGrid (Twilio Inc.) for transactional emails, and Microsoft Ireland Operations Ltd. (Azure OpenAI Service) for AI features when Vantrel Assist is enabled.

• ↳ Sub-processors › Infrastructure sub-processors
• ↳ Sub-processors › Service sub-processors
• ↳ Sub-processors › Notes (not in expected sources)

Reference answer

AWS (hosting), Cloudflare (CDN/WAF/DDoS), Datadog (monitoring), Stripe (billing), Zendesk (support), SendGrid/Twilio (transactional emails), Microsoft Azure OpenAI Service (AI features).

Expected: Subprocessors › Infrastructure sub-processors · Subprocessors › Service sub-processors

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer aligns closely with the reference answer, listing all the same sub-processors and their primary purposes. The assistant's answer includes additional details (e.g., specific legal entities like 'Amazon Web Services EMEA SARL' or 'Stripe Payments Europe, Ltd.') and clarifies data locations, which are not strictly necessary but do not detract from correctness. The assistant also correctly notes the conditional use of Microsoft Azure OpenAI Service (only when Vantrel Assist is enabled), which matches the reference. The cited excerpts fully support all claims, and no hallucinations or unsupported extrapolations are present. The answer is complete, covering all key sub-processors and their roles as per the reference, with no omissions of critical details. The only minor difference is the inclusion of legal entity names, which are not in the reference but do not affect correctness or completeness.

Drafted answer

Customer data is never used to train or fine-tune AI models, neither by Vantrel nor by its AI sub-processor.

• ↳ AI Features Policy › Use of customer data

Reference answer

No. Customer data is never used to train or fine-tune AI models, neither by Vantrel nor by its AI sub-processor.

Expected: Ai features policy › Use of customer data

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is factually consistent with the reference answer, which states that customer data is never used to train or fine-tune AI models by Vantrel or its sub-processor. The cited excerpts directly support this claim, and the assistant does not introduce any unsupported additions or omissions. The answer is concise and aligns perfectly with the reference, covering all key points without unnecessary details or inaccuracies.

Drafted answer

Yes, we support Single Sign-On via OpenID Connect and SAML 2.0 on Business and Enterprise plans, and Okta is one of the tested identity providers.

• ↳ Identity and Access Management › Customer authentication

Reference answer

Yes, via SAML 2.0 or OIDC SSO on Business and Enterprise plans; Okta is a tested identity provider.

Expected: Identity access › Customer authentication

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly states that employees can sign in with their corporate Okta accounts via SAML 2.0 and OIDC on Business and Enterprise plans, aligning perfectly with the reference answer. The cited excerpts explicitly mention Okta as a tested identity provider and confirm SSO support for those plans. The assistant’s phrasing is precise and avoids unsupported claims or omissions, covering all key points from the reference. The only minor difference is the assistant’s phrasing of 'we support' instead of 'yes,' which is semantically equivalent and does not affect correctness or completeness.

Drafted answer

In the event of a major incident affecting an entire AWS region, our recovery point objective is 1 hour.

• ↳ Business Continuity and Disaster Recovery › Objectives

Reference answer

At most about 1 hour of data (RPO 1 hour), with service restored from the Frankfurt DR region within 4 hours (RTO).

Expected: Business continuity › Objectives · Infrastructure hosting › Hosting regions

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies the **Recovery Point Objective (RPO)** of **1 hour** as stated in the cited excerpt, which aligns with the reference answer. However, the assistant omits the **Recovery Time Objective (RTO)** of **4 hours**, which is a key point in the reference answer. While the assistant's response is factually correct for the RPO, it fails to address the RTO, making it incomplete. The cited excerpt fully supports the RPO claim, so faithfulness is high, but completeness is reduced due to the omission of the RTO. The assistant did not hallucinate or misrepresent any facts, so correctness is high for the provided information, though the answer is not exhaustive.

Drafted answer

Vantrel employees do not have standing access to customer data. Support engineers can only access a customer workspace when the customer grants temporary access from the administration console.

• ↳ Identity and Access Management › Vantrel employee access to customer data

Reference answer

No standing access. Support can only access a workspace when the customer grants temporary access (max 72 hours), which is logged and visible in the audit trail.

Expected: Identity access › Vantrel employee access to customer data

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer correctly states that Vantrel employees lack standing access to customer data and that support engineers require temporary access granted by the customer. However, the assistant's answer omits the **time limit (72 hours)** and the **audit trail visibility** mentioned in both the reference and the cited excerpts, which are key details. While the core point (no standing access) is correct, the omission of these qualifications reduces the score for completeness. The assistant's claims are fully supported by the cited excerpts, so faithfulness is high. The reference answer also includes the time limit and audit trail, which the assistant missed, further justifying the lower completeness score.

Drafted answer

Enterprise customers can provide and control their own encryption key through AWS KMS.

• ↳ Security Whitepaper › Encryption

Reference answer

Yes, Enterprise customers can use customer-managed keys (BYOK) via AWS KMS; revoking the key makes the data unreadable by Vantrel.

Expected: Security whitepaper › Encryption

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies that Enterprise customers can use customer-managed keys (BYOK) via AWS KMS, which aligns with the reference answer. However, the assistant's response omits a critical detail from the reference: **revoking the key makes the data unreadable by Vantrel**, which is a key point about the implications of BYOK. The cited excerpts fully support the assistant's claim, but the answer lacks this critical qualification. The reference answer is fully covered by the excerpts, but the assistant's response is incomplete due to the missing detail about data unreadability upon revocation.

Drafted answer

We perform background checks on all new employees before their start date, where permitted by local law.

• ↳ Security Whitepaper › Employee security

Reference answer

Yes, background checks are performed on all new employees before their start date, where permitted by local law.

Expected: Security whitepaper › Employee security

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer directly matches the reference answer in both content and phrasing, confirming that background checks are performed on all new employees before their start date, where permitted by local law. The cited excerpt fully supports this claim, and no additional unsupported statements or omissions are present. The assistant did not introduce any incorrect facts or extrapolate beyond the documentation. Since the reference answer is fully aligned with the assistant's response, all criteria are met at the highest level.

Drafted answer

For Business customers, response time for a P1 service unavailable issue is 4 hours during business hours from Monday to Friday. For Enterprise customers, support is available 24/7 for P1 issues with a 1-hour first-response target. Starter customers have access to email support on a best-effort basis.

• ↳ Service Level Agreement and Support › Support

Reference answer

Enterprise: P1 first response within 1 hour, 24/7. Business: support is business hours only, P1 first response within 4 hours.

Expected: Sla support › Support

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer aligns perfectly with the reference answer for both Business and Enterprise tiers, correctly stating the response times (4 hours for Business during business hours, 1 hour for Enterprise 24/7) and explicitly noting the 'best-effort' basis for Starter customers. The cited excerpts fully support these claims, including the table and the additional note about Starter customers. The answer is concise, covering all key points without unnecessary details or inaccuracies, and no contradictions or omissions are present.

Drafted answer

Yes, some of our sub-processors process data in the US or globally. Specifically, SendGrid processes data in the US, Stripe processes data in the EU or US, AWS offers an option for US hosting depending on customer selection, Cloudflare processes traffic in transit globally, and Datadog processes logs and monitoring data in the US for US tenants.

• ↳ Sub-processors › Service sub-processors
• ↳ Sub-processors › Infrastructure sub-processors

Reference answer

For EU tenants, hosting (AWS), monitoring (Datadog), support (Zendesk) and AI (Azure EU Data Zone) stay in the EU. Cloudflare handles traffic in transit on its global network, Stripe operates in EU/US, and SendGrid processes email addresses and notification content in the US. Transfers are covered by SCCs; Vantrel Inc. is under the EU-U.S. Data Privacy Framework.

Expected: Subprocessors › Infrastructure sub-processors · Subprocessors › Service sub-processors · Data processing agreement › International data transfers

Grader · correctness 2 · faithfulness 2 · completeness 3

The assistant's answer incorrectly states that **AWS offers an option for US hosting depending on customer selection** without qualifying that this applies only to non-EU tenants (the reference specifies EU tenants use AWS in the EU). The assistant also **hallucinates** that Datadog processes logs in the US for EU tenants, which contradicts the excerpt (Datadog processes logs in the EU for EU tenants). While the assistant correctly identifies SendGrid, Stripe, and Cloudflare as processing data outside the EU, the unsupported claims about AWS and Datadog significantly undermine correctness and faithfulness. Completeness is partially met, but the incorrect details detract from the overall score.

Drafted answer

Customers affected by a security incident involving their data are notified by email by the Security team, in accordance with the timelines defined in the Data Processing Agreement. We notify the customer of any personal data breach affecting their data without undue delay, and in any case within 48 hours after becoming aware of it. Additionally, service availability incidents are published on the public status page at status.vantrel.example.

• ↳ Incident Response › Customer communication
• ↳ Data Processing Agreement (summary) › Personal data breach notification

Reference answer

Availability incidents on the status page (updates at least every 60 min for SEV1); customers whose data is affected are emailed by Security within the DPA timeline (48 hours); a root cause analysis is shared within 10 business days after a SEV1.

Expected: Incident response › Customer communication · Data processing agreement › Personal data breach notification

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant's answer aligns closely with the reference answer in terms of correctness, covering the email notification for affected customers within 48 hours (DPA timeline) and the status page updates for availability incidents. However, the assistant omitted the **root cause analysis (RCA) sharing within 10 business days for SEV1 incidents**, which is a key point in the reference. The cited excerpts fully support the assistant's claims, but the omission of the RCA detail reduces completeness. The assistant did not hallucinate or extrapolate unsupported information, and the cited documentation matches the reference answer where applicable. The assistant’s phrasing ('without undue delay, and in any case within 48 hours') is slightly redundant but does not introduce errors.

Drafted answer

Customer users can enable MFA using a TOTP authenticator app or a FIDO2/WebAuthn security key, and administrators can make MFA mandatory for all users of their workspace, while SMS-based MFA is not supported. The provided documentation does not contain information about MFA options for Vantrel staff accessing production.

• ↳ Identity and Access Management › Customer authentication

Reference answer

Customers: TOTP or FIDO2/WebAuthn, can be made mandatory by admins, no SMS. Staff: production admin access goes through a bastion with SSO, MFA and session recording; direct SSH disabled.

Expected: Identity access › Customer authentication · Infrastructure hosting › Network security

Grader · correctness 3 · faithfulness 3 · completeness 3

The assistant's answer correctly describes the MFA options for customer users (TOTP, FIDO2/WebAuthn, and mandatory enforcement by admins) and accurately states that SMS-based MFA is not supported, all of which align with the reference and cited excerpts. However, the assistant claims the documentation does not contain information about MFA options for Vantrel staff accessing production, which is incorrect-the reference explicitly states that staff production access requires a bastion with SSO, MFA, and session recording, with direct SSH disabled. This omission of key details for staff access reduces correctness and completeness scores. Faithfulness is high since all customer-related claims are supported by the excerpts, but the unsupported claim about staff MFA documentation hurts the score. Completeness is also reduced due to the missing staff-specific details.

Drafted answer

Infrastructure and application logs are centralized in Datadog, specifically on the EU site for EU-hosted tenants and the US site for US-hosted tenants. Infrastructure logs are retained for 12 months, and user activity audit logs are retained for 13 months, but the documentation does not state the retention period for application logs.

• ↳ Infrastructure and Hosting › Logging and monitoring
• ↳ Identity and Access Management › Audit logs
• ↳ Sub-processors › Infrastructure sub-processors (not in expected sources)

Reference answer

Infrastructure and application logs are centralized in Datadog (EU site for EU tenants), retained 12 months. Customer audit logs are retained 13 months and exportable via API or streamed to a SIEM (Enterprise).

Expected: Infrastructure hosting › Logging and monitoring · Identity access › Audit logs

Grader · correctness 1 · faithfulness 3 · completeness 3

The assistant's answer contains a **correctness** issue by stating that application logs are stored in both EU and US Datadog sites, which contradicts the reference answer (only EU for EU tenants). It also incorrectly claims the documentation does not specify the retention period for application logs, while the reference explicitly states they are retained for 12 months. The **faithfulness** score is 3 because the assistant extrapolates the US site for application logs from the sub-processor table (which only mentions *monitoring, logs* generically) without explicit confirmation. The **completeness** score is 3 because it omits the exportability of audit logs (API/SIEM) and the 13-month retention for customer audit logs, though it covers the main points of infrastructure logs and retention periods (with errors).

Drafted answer

When Vantrel Assist is enabled, data is sent to our AI sub-processor, Microsoft Ireland Operations Ltd. (Azure OpenAI Service), which is deployed in the EU Data Zone for EU-hosted tenants. Only the data required to answer a specific request is sent to the model.

• ↳ Sub-processors › Service sub-processors
• ↳ AI Features Policy › Vantrel Assist
• ↳ Sub-processors › Notes (not in expected sources)

Reference answer

Only if a workspace Admin enables Vantrel Assist (off by default). Then only the data needed for the request is sent to Microsoft Azure OpenAI Service in the EU Data Zone; it is never used for training.

Expected: Ai features policy › Vantrel Assist · Ai features policy › Use of customer data · Subprocessors › Service sub-processors

Grader · correctness 5 · faithfulness 5 · completeness 4

The assistant's answer correctly identifies that data is sent to Microsoft Ireland Operations Ltd. (Azure OpenAI Service) in the EU Data Zone when Vantrel Assist is enabled, aligning with the reference answer. However, the assistant's phrasing ('our AI sub-processor') is slightly less precise than the reference ('Microsoft Azure OpenAI Service'), though this does not affect correctness. The assistant's answer is fully supported by the cited excerpts and covers all key points: the conditional nature of data transfer (only when Vantrel Assist is enabled), the specific sub-processor (Microsoft Ireland Operations Ltd.), and the data location (EU Data Zone). The assistant does not mention the 'off by default' status or the explicit exclusion of training, but these are not explicitly required in the question and are secondary details in the reference. The assistant's answer is faithful to the excerpts and does not introduce unsupported claims or hallucinations.

Drafted answer

We run a private bug bounty program with invited researchers, but our documentation does not disclose reward amounts.

• ↳ Incident Response › Reporting a vulnerability

Reference answer

Yes, a private bug bounty program with invited researchers. Reward amounts are not documented.

Expected: Incident response › Reporting a vulnerability

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly identifies the existence of a private bug bounty program with invited researchers, which aligns perfectly with the reference answer and the cited documentation. However, the assistant's claim that 'reward amounts are not documented' is accurate and consistent with the reference answer, even though the documentation excerpt does not explicitly mention reward amounts (it is implied by the reference). The assistant's reasoning is faithful to the provided excerpts and covers all key points from the reference answer, including the lack of documentation on reward amounts. The assistant did not hallucinate or infer unsupported details beyond what is stated or implied in the documentation or reference.

Drafted answer

We are tested by an independent third-party security firm at least once a year, but the name of the specific firm that performed the penetration test is not documented.

• ↳ Penetration Testing and Vulnerability Management › Penetration testing

Reference answer

An independent third-party security firm, in March 2026. The firm's name is not documented.

Expected: Vulnerability management › Penetration testing

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies that the penetration test was performed by an independent third-party firm and that the name of the firm is not documented, aligning with the reference answer. However, the assistant incorrectly generalizes the frequency of testing ('at least once a year') without explicitly confirming the March 2026 date, which is a key fact in the reference. The assistant's claim about annual testing is supported by the excerpts, but the omission of the specific date (March 2026) and the lack of mention of the grey-box approach, scope (web app, API, infrastructure), or findings (no critical, two high-severity) reduces completeness. The assistant’s answer is faithful to the excerpts for the parts it covers but misses critical details present in the reference and excerpts. The assistant does not hallucinate or introduce unsupported claims beyond the minor extrapolation of frequency.

Drafted answer

Enterprise customers can stream audit logs to their SIEM via webhook.

• ↳ Identity and Access Management › Audit logs

Reference answer

Enterprise customers can stream audit logs to their SIEM via webhook; specific supported SIEM products are not listed.

Expected: Identity access › Audit logs

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly states that enterprise customers can stream audit logs to their SIEM via webhook, which aligns perfectly with the reference answer. The reference explicitly mentions that no specific SIEM products are listed, and the assistant does not claim otherwise, avoiding incorrect assumptions. The cited excerpt directly supports the assistant's statement, and the answer covers the key point of the reference without omissions or unsupported claims. Since the reference confirms the information is not documented beyond the general capability, the assistant's response is accurate and complete.

Drafted answer

Our Data Protection Officer can be contacted at dpo@vantrel.example.

• ↳ Company Overview › Security and compliance organization

Reference answer

The DPO can be contacted at dpo@vantrel.example; the DPO's name is not documented.

Expected: Company overview › Security and compliance organization

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies the contact email for the Data Protection Officer (DPO) as `dpo@vantrel.example`, which matches the reference answer and is explicitly stated in the cited excerpt. However, the assistant omits the key point from the reference answer that the DPO's name is **not documented**, which is a critical detail for completeness. The assistant’s answer is factually correct but incomplete. The cited excerpt fully supports the email address provided, so faithfulness is high. The omission of the 'name not documented' detail reduces the completeness score significantly, as it is a key point in the reference answer.

Drafted answer

Custom roles are not currently supported by Vantrel.

• ↳ Identity and Access Management › Roles and permissions

Reference answer

No, custom roles are not currently supported. Four default roles exist (Admin, Modeler, Contributor, Viewer) plus dimension-member-level restrictions.

Expected: Identity access › Roles and permissions

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is factually consistent with the reference answer, as both state that custom roles are not supported. The assistant's response is fully supported by the cited excerpt, which explicitly mentions 'Custom roles are not currently supported.' The answer is also complete, as it covers the key point of the reference without omitting any critical details or adding unsupported information. The assistant correctly aligns with the documentation and does not introduce any inaccuracies or hallucinations.

Drafted answer

No, we do not offer on-premise deployments as Vantrel Plan is delivered exclusively as a multi-tenant Software-as-a-Service application.

• ↳ Company Overview › Product and delivery model

Reference answer

No. Vantrel Plan is SaaS only; no on-premise, private cloud or self-hosted deployment.

Expected: Company overview › Product and delivery model

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is factually consistent with the reference answer, correctly stating that Vantrel Plan is SaaS-only with no on-premise, private cloud, or self-hosted options. The cited excerpt explicitly supports this claim, and the assistant's phrasing ('delivered exclusively as a multi-tenant SaaS application') aligns perfectly with the reference. The answer is concise, covers all key points (no deployment alternatives), and avoids unsupported additions or omissions. Since the reference confirms the information is documented, the assistant’s response is both faithful and complete.

Drafted answer

No, SMS-based MFA is not supported.

• ↳ Identity and Access Management › Customer authentication

Reference answer

No, SMS-based MFA is not supported (TOTP and FIDO2/WebAuthn are).

Expected: Identity access › Customer authentication

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is factually correct and aligns perfectly with the reference answer, which explicitly states that SMS-based MFA is not supported. The assistant cited the correct excerpt from the documentation, which directly confirms the absence of SMS-based MFA support and lists the supported alternatives (TOTP and FIDO2/WebAuthn). The answer is concise, covers the key point, and does not include any unsupported claims or omissions. Since the reference answer and the assistant's answer are identical in substance, all criteria are fully satisfied.

Drafted answer

No, data cannot be hosted in Australia because no other hosting regions are currently offered besides the EU and US.

• ↳ Infrastructure and Hosting › Hosting regions

Reference answer

No. Only EU (Paris/Frankfurt) and US (N. Virginia/Oregon) regions are offered.

Expected: Infrastructure hosting › Hosting regions

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly aligns with the reference answer, stating that data cannot be hosted in Australia and only EU and US regions are offered. The cited excerpts explicitly list the available regions (EU and US) and confirm no other regions are currently available, directly supporting the assistant's claim. The answer is concise, avoids unsupported extrapolations, and covers the key point of regional restrictions without omissions or contradictions.

Drafted answer

Vantrel is working toward conformance with WCAG 2.1 level AA, but an accessibility conformance report (VPAT) is not yet available.

• ↳ Product FAQ › Is Vantrel Plan accessible to users with disabilities?

Reference answer

Not yet. Vantrel is working toward WCAG 2.1 AA, but a VPAT is not available yet.

Expected: Product faq › Is Vantrel Plan accessible to users with disabilities?

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is factually consistent with the reference answer, as both state that Vantrel is working toward WCAG 2.1 AA but lacks a VPAT. The cited excerpt directly supports the assistant's claims, and no key points are omitted. The phrasing is nearly identical, ensuring correctness, faithfulness, and completeness.

Drafted answer

The provided documentation does not contain information about FedRAMP authorization.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 3 · completeness 5

The assistant correctly identifies that the information about FedRAMP authorization for Vantrel Plan is not documented, which aligns perfectly with the reference answer. However, the assistant did not cite any excerpts to support this claim, which is a minor but notable omission since the question explicitly requires referencing documentation. The answer is factually correct, complete, and faithful to the reference (as the reference itself states the information is not documented), but the lack of cited excerpts slightly detracts from the faithfulness score due to procedural expectations.

Drafted answer

The provided documentation does not contain information regarding whether Vantrel will sign a HIPAA Business Associate Agreement (BAA).

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the documentation does not address the question of signing a HIPAA BAA, aligning perfectly with the reference answer. However, the assistant did not cite any excerpts, which is a minor procedural issue but does not affect the correctness or completeness of the response. Since the reference explicitly states the information is not documented, the assistant’s answer is fully accurate and complete.

Drafted answer

The provided documentation does not contain information about the coverage amount of our cyber insurance policy.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 3 · completeness 5

The assistant correctly identifies that the coverage amount of the cyber insurance policy is not documented, which aligns perfectly with the reference answer. However, the assistant did not cite any excerpts to support this claim, which is unusual since the reference answer explicitly states the information is not documented. Since the reference confirms the absence of documentation, the assistant’s answer is factually correct, but the lack of cited excerpts slightly detracts from faithfulness. Completeness is fully satisfied as the answer directly addresses the question without omissions or additions.

Drafted answer

The provided documentation does not contain information regarding source code escrow agreements.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the documentation does not address the question of a source code escrow agreement, which aligns perfectly with the reference answer. However, the assistant did not cite any excerpts, which is a minor procedural issue but does not affect the correctness or completeness of the response. Since the reference explicitly states the information is not documented, the answer is fully accurate and complete. Faithfulness is not applicable here because no claims are made beyond the absence of documentation, and no excerpts were required to support this conclusion.

Drafted answer

The provided documentation does not contain information regarding IP allowlisting to restrict user access.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the documentation does not address IP allowlisting for restricting user access, which aligns perfectly with the reference answer. Since the reference explicitly states the information is not documented, the assistant’s response is accurate and complete. However, the assistant did not cite any excerpts, which is irrelevant in this case because the reference confirms the absence of documentation, but it would have been better practice to explicitly state this (e.g., 'No excerpts were found').

Drafted answer

The provided documentation does not contain information about post-quantum cryptography.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 3 · completeness 5

The assistant correctly identifies that the documentation does not address post-quantum cryptography, aligning perfectly with the reference answer. However, the assistant did not cite any excerpts, which is a minor procedural issue but does not affect the correctness or completeness of the response. Since the reference explicitly states the information is not documented, the answer is fully accurate and complete, though the lack of cited excerpts slightly impacts the faithfulness score due to procedural expectations.

Drafted answer

The provided documentation does not contain information about a SOC 1 report.

Reference answer

Not in the documentation. Only Vantrel's SOC 2 Type II is documented; the SOC 1 mentioned is AWS's, not Vantrel's.

Grader · correctness 5 · faithfulness 3 · completeness 3

The assistant correctly identifies that the SOC 1 report is not documented, which aligns perfectly with the reference answer. However, the assistant did not cite any excerpts to support this claim, which is a minor but notable omission since the reference explicitly mentions AWS's SOC 1 (not Vantrel's) and Vantrel's SOC 2 Type II. The absence of citations means the answer lacks direct evidence, though the factual accuracy is correct. Completeness is high since the core point (no SOC 1 report in Vantrel's documentation) is addressed, but the reference's nuance about AWS's SOC 1 is missing.

Drafted answer

The provided documentation does not contain information about the number of people working in our security team.

Reference answer

Not in the documentation. Only the total headcount (~240) and the CISO/Security Committee are mentioned.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the documentation does not provide the specific number of people in the security team, aligning perfectly with the reference answer. However, the assistant did not cite any excerpts to support this claim, which is unnecessary in this case since the reference explicitly states the information is not documented. The answer is concise, accurate, and covers the key point of the reference without omissions or additions.

Drafted answer

We do not have a mean time to resolve P1 incidents, as the documentation only defines first-response targets and states that these are not resolution times.

• ↳ Service Level Agreement and Support › Support

Reference answer

Not in the documentation. Only first-response targets are documented, explicitly not resolution times.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly aligns with the reference answer by stating that the mean time to resolve P1 incidents is not documented, and it explicitly cites the documentation's clarification that the provided times are *first-response targets* and not resolution times. The assistant's reasoning is fully supported by the cited excerpts, which directly state this limitation. The answer is also complete, as it covers the key point that resolution times are not documented, and it avoids introducing unsupported claims or omitting critical details from the reference. The assistant did not hallucinate or infer beyond the provided documentation.

Drafted answer

The provided documentation does not contain information about CSA STAR certification.

Reference answer

Not in the documentation. The documented certifications are ISO 27001 and SOC 2 Type II.

Grader · correctness 5 · faithfulness 3 · completeness 5

The assistant correctly identifies that the information about CSA STAR certification is not documented, which aligns perfectly with the reference answer. However, the assistant did not cite any excerpts to support this claim, which is a minor but notable omission since the reference answer explicitly states that the documented certifications are ISO 27001 and SOC 2 Type II. The absence of cited excerpts prevents a full score for faithfulness, though the correctness and completeness are flawless given the context of the question and reference answer.

Drafted answer

The provided documentation does not contain information about an emergency phone number to call during a security incident.

Reference answer

Not in the documentation. Only email (security@vantrel.example) and the status page are documented.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the documentation does not provide an emergency phone number, aligning perfectly with the reference answer. However, the assistant did not cite any excerpts to support this claim, which is a minor but notable omission since the reference explicitly states the absence of such information in the documentation. The answer is concise and avoids incorrect assumptions or unsupported claims, focusing solely on the documented communication channels (email and status page).

Drafted answer

According to our Security Whitepaper, customer data is permanently deleted from production systems within 90 days of contract termination, while our Data Processing Agreement states it is deleted within 30 days after the end of the 30-day grace period. Backups containing the data expire according to the backup retention schedule.

• ↳ Security Whitepaper › Data retention and deletion
• ↳ Data Processing Agreement (summary) › Return and deletion of data

Reference answer

The documentation is inconsistent: the Security Whitepaper says within 90 days of termination, while the DPA says within 30 days after a 30-day export grace period (i.e. ~60 days). Both sources should be cited and the inconsistency flagged.

Expected: Security whitepaper › Data retention and deletion · Data processing agreement › Return and deletion of data

Grader · correctness 3 · faithfulness 3 · completeness 3

The assistant correctly identified the inconsistency between the **Security Whitepaper** (90 days) and the **DPA** (30 days after a 30-day grace period, effectively ~60 days). However, the assistant did not explicitly state the **~60-day total** implied by the DPA (30-day grace + 30-day deletion window), which weakens completeness. The assistant also added an unsupported claim about backups expiring *according to the backup retention schedule* without citing a specific source, which slightly reduces faithfulness. The core facts (90 days vs. ~60 days) are correct and supported by the excerpts, but the phrasing could be more precise regarding the DPA’s timeline. The assistant’s answer is mostly faithful and complete but misses a minor refinement in clarity and omits the legal retention caveat from the DPA.

Drafted answer

Pour les clients de la region EU, la localisation principale des donnees est AWS eu-west-3 a Paris, en France. Aucun autre site d'hebergement n'est propose.

• ↳ Infrastructure and Hosting › Hosting regions

Reference answer

Oui pour la région EU (par défaut) : AWS eu-west-3 à Paris, avec reprise d'activité à Francfort (Allemagne). Une région US existe aussi.

Expected: Infrastructure hosting › Hosting regions

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies the primary hosting location for EU clients as AWS eu-west-3 in Paris, France, which aligns with the reference answer. However, the assistant omits the disaster recovery location (Frankfurt, Germany) and the existence of a US region, which are key points in the reference. Additionally, the assistant claims 'Aucun autre site d'hébergement n'est propose' (no other hosting sites are offered), which is technically correct but does not mention the US region explicitly, leading to a lack of completeness. The cited excerpts fully support the assistant's claim about the EU region's primary location, but the omission of critical details reduces the completeness score.

Drafted answer

Yes, SAML 2.0 and OpenID Connect (OIDC) Single Sign-On is supported on Business and Enterprise plans, and Microsoft Entra ID is among the tested identity providers.

• ↳ Identity and Access Management › Customer authentication

Reference answer

Oui, SSO SAML 2.0 et OIDC sur les plans Business et Enterprise ; Microsoft Entra ID fait partie des fournisseurs testés.

Expected: Identity access › Customer authentication

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is factually consistent with the reference answer, correctly stating that SAML 2.0 and OIDC SSO are supported on Business and Enterprise plans and that Microsoft Entra ID is among the tested identity providers. The cited excerpts fully support these claims, and the answer covers all key points from the reference. No contradictions or omissions are present, and the assistant did not introduce unsupported details or extrapolations. The reference explicitly confirms the information, so no additional qualifications or warnings are needed beyond what is stated.

Drafted answer

The provided documentation does not contain information regarding SecNumCloud certification by ANSSI.

Reference answer

Pas dans la documentation.

Grader · correctness 5 · faithfulness 3 · completeness 5

The assistant correctly identifies that the documentation does not address the question of SecNumCloud qualification by ANSSI, aligning perfectly with the reference answer. However, the assistant did not cite any excerpts, which is a minor procedural issue but does not affect the correctness or completeness of the response. Since the reference explicitly states the information is not documented, the answer is fully accurate and complete, though the lack of cited excerpts slightly impacts faithfulness (as no excerpts were provided to verify).