Drafted by Gemini, graded by Mistral

Drafted answer

All customer data stored in databases, file storage, and backups is encrypted at rest using AES-256. Keys are managed in AWS Key Management Service and are rotated automatically every year.

• ↳ Security Whitepaper › Encryption

Reference answer

Yes. All customer data (databases, file storage, backups) is encrypted at rest with AES-256, keys managed in AWS KMS and rotated automatically every year.

Expected: Security whitepaper › Encryption

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is **fully consistent** with the reference answer in terms of correctness, as it accurately states that customer data (databases, file storage, backups) is encrypted at rest using AES-256, with keys managed in AWS KMS and rotated annually. The cited excerpts **directly support** all claims, and the answer **covers all key points** without omissions or unsupported additions. The reference does not mention any contradictions or missing qualifications, and the assistant did not introduce any incorrect or extraneous details. The only minor deviation is the inclusion of the 'Customer-managed keys (BYOK)' excerpt, which is irrelevant to the question but does not detract from the score since it does not affect correctness, faithfulness, or completeness for the asked question.

Drafted answer

All connections to Vantrel Plan use TLS 1.2 or higher, with TLS 1.3 preferred and negotiated when supported by the client. Older protocols such as SSL, TLS 1.0, and 1.1 are disabled.

• ↳ Security Whitepaper › Encryption

Reference answer

TLS 1.2 is the minimum; TLS 1.3 is preferred. SSL, TLS 1.0 and 1.1 are disabled, and HSTS is enforced.

Expected: Security whitepaper › Encryption

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer aligns closely with the reference answer, correctly stating that TLS 1.2 is the minimum supported version, TLS 1.3 is preferred, and older protocols (SSL, TLS 1.0, and TLS 1.1) are disabled. The assistant also accurately includes the enforcement of HSTS, which is a key point in the reference. The cited excerpts fully support all claims made in the answer, and no additional unsupported details are included. The answer is complete, covering all key points from the reference without omissions or extraneous information.

Drafted answer

Yes, Vantrel is ISO/IEC 27001:2022 certified, and has been certified since September 2023.

• ↳ Security Whitepaper › Certifications and audit reports

Reference answer

Yes, ISO/IEC 27001:2022 certified since September 2023, covering development, operation and support of Vantrel Plan, with annual surveillance audits.

Expected: Security whitepaper › Certifications and audit reports

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant's answer correctly states that Vantrel is ISO/IEC 27001:2022 certified since September 2023, which aligns perfectly with the reference answer. However, the assistant's answer omits the **scope of certification** (development, operation, and support of Vantrel Plan) and **annual surveillance audits**, which are key points in the reference. The cited excerpts fully support the assistant's claims, but the omission of these details affects completeness. The assistant did not hallucinate or introduce unsupported claims, and the answer is faithful to the documentation provided. The correctness is slightly reduced due to the omission of scope details, but the core fact (certification status and date) is accurate.

Drafted answer

Yes, we undergo an annual SOC 2 Type II examination covering the Security, Availability, and Confidentiality Trust Services Criteria. The report can be shared with customers and prospects under a non-disclosure agreement on request to security@vantrel.example.

• ↳ Data Processing Agreement (summary) › Audits (not in expected sources)
• ↳ Security Whitepaper › Certifications and audit reports

Reference answer

Yes. Vantrel has an annual SOC 2 Type II (Security, Availability, Confidentiality; period Jan 1 - Dec 31, report available around March). It is shared under NDA on request to security@vantrel.example.

Expected: Security whitepaper › Certifications and audit reports

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer is **correct** as it aligns with the reference answer in stating that a SOC 2 Type II report exists, covers Security, Availability, and Confidentiality, and is shared under NDA upon request. The **faithfulness** is high since all claims are directly supported by the cited excerpts, particularly the Security Whitepaper. However, **completeness** is slightly lacking: the assistant omits the specific audit period (Jan 1-Dec 31) and the approximate availability date (March of the following year), which are explicitly mentioned in the reference and excerpts. These are key details that should be included for full completeness. The assistant’s answer does not introduce any incorrect or unsupported claims, and the cited documentation fully backs the provided information.

Drafted answer

For EU-hosted tenants, customer data is stored and processed in the European Union, specifically at AWS eu-west-3 in Paris, France, with disaster recovery at AWS eu-central-1 in Frankfurt, Germany. Customer data at rest stays within the selected region, including backups and disaster recovery copies.

• ↳ Data Processing Agreement (summary) › International data transfers (not in expected sources)
• ↳ Infrastructure and Hosting › Hosting regions

Reference answer

On AWS in eu-west-3 (Paris, France), with disaster recovery in eu-central-1 (Frankfurt, Germany). Data at rest, including backups, stays in the selected region.

Expected: Infrastructure hosting › Hosting regions

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is highly accurate and aligns perfectly with the reference answer, covering all key points: the primary hosting region (AWS eu-west-3 in Paris), the disaster recovery location (AWS eu-central-1 in Frankfurt), and the fact that data at rest (including backups) remains within the selected region. The assistant's phrasing is consistent with the reference, and all claims are directly supported by the cited excerpts, particularly the Infrastructure and Hosting table. The only minor difference is the assistant's inclusion of 'customer data at rest stays within the selected region, including backups and disaster recovery copies,' which is explicitly stated in the excerpts and thus fully justified. No contradictions or omissions are present.

Drafted answer

For Vantrel Plan, our Recovery Point Objective is 1 hour and our Recovery Time Objective is 4 hours in the event of a major incident affecting an entire region.

• ↳ Business Continuity and Disaster Recovery › Objectives

Reference answer

RPO of 1 hour and RTO of 4 hours.

Expected: Business continuity › Objectives

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly identifies the RPO and RTO for the Vantrel Plan as 1 hour and 4 hours, respectively, which aligns perfectly with the reference answer. The cited excerpts explicitly state these values, ensuring both correctness and faithfulness. The assistant also correctly specifies the context (major incident affecting an entire AWS region), which is consistent with the documentation. The answer is concise and covers all key points without unnecessary details, making it fully complete.

Drafted answer

Databases are backed up continuously with point-in-time recovery available for the last 7 days, and full encrypted snapshots are taken daily and retained for 35 days. Backups containing customer data expire according to this backup retention schedule after contract termination.

• ↳ Business Continuity and Disaster Recovery › Backups
• ↳ Security Whitepaper › Data retention and deletion (not in expected sources)

Reference answer

Continuous backups with 7-day point-in-time recovery, plus daily encrypted full snapshots retained for 35 days, replicated to the DR location. Restores are tested monthly.

Expected: Business continuity › Backups

Grader · correctness 3 · faithfulness 3 · completeness 3

The assistant's answer correctly captures the core backup frequency (continuous with 7-day point-in-time recovery and daily encrypted snapshots retained for 35 days) and aligns with the reference. However, it omits the **replication to the DR location** (a key point in the reference) and incorrectly states that backups expire *according to the backup retention schedule* after contract termination, while the reference (and cited excerpts) clarify that **customer data is permanently deleted from production systems within 90 days** of termination, with backups expiring per the retention schedule (35 days). The assistant’s addition about contract termination is unsupported by the cited excerpts (it conflates backup retention with data deletion timelines). The reference also emphasizes **monthly restore testing**, which the assistant omits entirely. The cited excerpts fully support the backup frequency/retention claims but lack the DR replication detail and restore testing note from the reference.

Drafted answer

We support SAML 2.0 and OpenID Connect (OIDC) single sign-on protocols on Business and Enterprise plans.

• ↳ Identity and Access Management › Customer authentication

Reference answer

SAML 2.0 and OpenID Connect, on Business and Enterprise plans. Tested with Microsoft Entra ID, Okta and Google Workspace. Enforcing SSO disables password login.

Expected: Identity access › Customer authentication

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies the supported SSO protocols (SAML 2.0 and OpenID Connect) and the plan restrictions (Business and Enterprise). However, it omits key details from the reference answer: the tested identity providers (Microsoft Entra ID, Okta, Google Workspace) and the enforcement behavior (disabling password login when SSO is enforced). While the assistant's answer is factually correct in its core points, it lacks critical context and qualifications present in the reference. The cited excerpts fully support the assistant's claims, but the omission of these details affects completeness. The assistant did not hallucinate or extrapolate unsupported information, so faithfulness remains high.

Drafted answer

We support user provisioning and deprovisioning through SCIM 2.0 on the Enterprise plan.

• ↳ Identity and Access Management › User provisioning
• ↳ Company Overview › Product and delivery model (not in expected sources)

Reference answer

Yes, SCIM 2.0 is supported, on the Enterprise plan only.

Expected: Identity access › User provisioning

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant's answer correctly states that SCIM 2.0 is supported for automated user provisioning, but it omits the critical detail that this feature is **Enterprise plan-only**, which is explicitly mentioned in both the reference answer and the cited excerpts. While the core fact (SCIM 2.0 support) is correct, the omission of the plan restriction reduces the correctness score. The assistant’s claims are fully supported by the cited excerpts, and the answer is concise but incomplete regarding the plan limitation. The reference answer’s key points are partially addressed, but the plan restriction is a critical detail missing in the assistant’s response.

Drafted answer

For Enterprise customers, we commit to a monthly uptime of 99.95%, excluding scheduled maintenance.

• ↳ Service Level Agreement and Support › Availability commitment

Reference answer

99.95% monthly uptime for Enterprise (excluding scheduled maintenance), with service credits of 10% or 25% of the monthly fee.

Expected: Sla support › Availability commitment

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer correctly states the **99.95% monthly uptime commitment for Enterprise customers**, which matches the reference answer. However, the reference also explicitly mentions **service credits of 10% or 25% of the monthly fee** in case of downtime, which the assistant omitted. The cited excerpts support the uptime percentage but do not mention service credits, so the assistant's answer is **partially incomplete** and **lacks a key detail** from the reference. The assistant did not hallucinate or misrepresent facts, but the omission of service credits reduces the completeness score.

Drafted answer

We notify the customer of any personal data breach affecting their data without undue delay, and in any case within 48 hours after becoming aware of it.

• ↳ Data Processing Agreement (summary) › Personal data breach notification

Reference answer

Without undue delay and in any case within 48 hours after becoming aware of it.

Expected: Data processing agreement › Personal data breach notification

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is factually consistent with the reference answer, as both state the notification must occur 'without undue delay' and 'within 48 hours' after becoming aware of the breach. The cited excerpt from the Data Processing Agreement directly supports the assistant's response, including the key timeframe and the obligation to notify. The answer is concise, covering all key points without unnecessary details or deviations.

Drafted answer

Vantrel Plan is tested by an independent third-party security firm at least once a year, and additionally before major architectural changes. The most recent penetration test was completed in March 2026 and identified no critical findings and two high-severity findings, both of which were remediated within 30 days and verified by the testing firm during a retest.

• ↳ Penetration Testing and Vulnerability Management › Penetration testing

Reference answer

At least annually by an independent third party, plus before major architectural changes. The last test (March 2026) found no critical and two high findings, both remediated within 30 days and verified by retest.

Expected: Vulnerability management › Penetration testing

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer aligns perfectly with the reference answer in terms of frequency (annual + pre-major architectural changes), timing (March 2026), and findings (no critical, two high-severity, remediated within 30 days with retest verification). The cited excerpts directly support every claim made, including the scope (web app, API, infrastructure) and methodology (grey-box). The assistant did not introduce unsupported details or omit key points, and the reference does not flag any contradictions. The answer is fully faithful, correct, and complete.

Drafted answer

We remediate critical vulnerabilities within 7 days.

• ↳ Penetration Testing and Vulnerability Management › Vulnerability management

Reference answer

Critical vulnerabilities must be remediated within 7 days (high: 30 days, medium: 90 days).

Expected: Vulnerability management › Vulnerability management

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer correctly states the remediation timeline for critical vulnerabilities as 7 days, which aligns perfectly with the reference answer and the cited documentation. However, the assistant's response is incomplete because it omits the timelines for high (30 days) and medium (90 days) vulnerabilities, which are key points in the reference and documentation. The cited excerpts fully support the assistant's statement about critical vulnerabilities, so faithfulness is not an issue here. The assistant did not hallucinate or add unsupported details, but the omission of other severity timelines affects completeness.

Drafted answer

Our sub-processors include Amazon Web Services EMEA SARL for cloud hosting, storage, and backups; Cloudflare, Inc. for CDN, Web Application Firewall, and DDoS protection; and Datadog, Inc. for infrastructure and application monitoring and logs. We also use Stripe Payments Europe, Ltd. for billing and subscription payments, Zendesk, Inc. for customer support ticketing, SendGrid for transactional emails, and Microsoft Ireland Operations Ltd. for AI features when Vantrel Assist is enabled.

• ↳ Sub-processors › Infrastructure sub-processors
• ↳ Sub-processors › Service sub-processors
• ↳ Sub-processors › Notes (not in expected sources)

Reference answer

AWS (hosting), Cloudflare (CDN/WAF/DDoS), Datadog (monitoring), Stripe (billing), Zendesk (support), SendGrid/Twilio (transactional emails), Microsoft Azure OpenAI Service (AI features).

Expected: Subprocessors › Infrastructure sub-processors · Subprocessors › Service sub-processors

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer aligns closely with the reference answer in terms of correctness, listing all the correct sub-processors and their primary purposes. The assistant's inclusion of specific legal entities (e.g., 'Amazon Web Services EMEA SARL' instead of just 'AWS') and additional granular details (e.g., 'when Vantrel Assist is enabled' for Microsoft Azure) is supported by the cited excerpts and does not contradict the reference. The assistant's answer is faithful to the excerpts, as every claim is directly supported by the provided documentation. The completeness is high, as all key sub-processors and their purposes are covered, with minor additional details that do not detract from the core accuracy. The only minor deviation is the phrasing of 'transactional emails' (reference) vs. 'transactional emails (notifications, password resets)' (assistant), which is still correct and supported by the excerpts.

Drafted answer

Customer data is never used to train or fine-tune AI models, neither by Vantrel nor by its AI sub-processor.

• ↳ AI Features Policy › Use of customer data

Reference answer

No. Customer data is never used to train or fine-tune AI models, neither by Vantrel nor by its AI sub-processor.

Expected: Ai features policy › Use of customer data

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is factually consistent with the reference answer, which states that customer data is never used to train or fine-tune AI models by Vantrel or its sub-processor. The cited excerpts directly support this claim, and the assistant does not introduce any unsupported additions or omissions. The answer is concise and aligns perfectly with the reference, covering all key points without unnecessary details or inaccuracies.

Drafted answer

Yes, Single Sign-On using OpenID Connect and SAML 2.0 is supported on Business and Enterprise plans, and tested identity providers include Okta.

• ↳ Identity and Access Management › Customer authentication

Reference answer

Yes, via SAML 2.0 or OIDC SSO on Business and Enterprise plans; Okta is a tested identity provider.

Expected: Identity access › Customer authentication

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly states that SSO via SAML 2.0 and OIDC is supported on Business and Enterprise plans, and explicitly mentions Okta as a tested identity provider, aligning perfectly with the reference answer. The cited excerpts fully support these claims, and no key points from the reference are missing. The assistant’s phrasing is precise and avoids unsupported extrapolations or hallucinations. The only minor difference is the reference’s use of 'Yes, via SAML 2.0 or OIDC SSO' vs. the assistant’s 'Yes, Single Sign-On using OpenID Connect and SAML 2.0,' which is functionally equivalent and does not detract from correctness or completeness. The excerpts also confirm the plan restrictions and the inclusion of Okta, fulfilling all criteria for a top score.

Drafted answer

In the event of a major incident affecting an entire primary region such as Paris, our Recovery Point Objective is 1 hour.

• ↳ Business Continuity and Disaster Recovery › Objectives

Reference answer

At most about 1 hour of data (RPO 1 hour), with service restored from the Frankfurt DR region within 4 hours (RTO).

Expected: Business continuity › Objectives · Infrastructure hosting › Hosting regions

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies the Recovery Point Objective (RPO) of 1 hour as stated in the cited excerpt, which aligns perfectly with the reference answer. However, the assistant omits the Recovery Time Objective (RTO) of 4 hours, which is a key point in the reference answer and the cited documentation. While the assistant's answer is factually correct for the RPO, it is incomplete as it does not address the RTO or the broader context of service restoration from the Frankfurt DR region. The cited excerpts fully support the RPO claim, but the omission of the RTO reduces the completeness score.

Drafted answer

Vantrel employees do not have standing access to customer data. Support engineers can only access a customer workspace when the customer grants temporary access from the administration console.

• ↳ Identity and Access Management › Vantrel employee access to customer data

Reference answer

No standing access. Support can only access a workspace when the customer grants temporary access (max 72 hours), which is logged and visible in the audit trail.

Expected: Identity access › Vantrel employee access to customer data

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer correctly states that Vantrel employees lack standing access to customer data and that support engineers require temporary access granted by the customer. However, the assistant's answer omits the **time limit (72 hours)** and the **audit trail visibility** mentioned in both the reference and the cited excerpts, which are key details. While the core point (no standing access) is correct, the omission of these qualifications reduces the score for completeness. The assistant's claims are fully supported by the cited excerpts, so faithfulness is high. The reference answer also includes the time limit and audit trail, which the assistant missed, further justifying the lower completeness score.

Drafted answer

Enterprise customers can provide and control their own encryption key through AWS KMS, allowing them to hold the keys to their own data where revoking the key makes the customer data unreadable by Vantrel.

• ↳ Security Whitepaper › Encryption

Reference answer

Yes, Enterprise customers can use customer-managed keys (BYOK) via AWS KMS; revoking the key makes the data unreadable by Vantrel.

Expected: Security whitepaper › Encryption

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer aligns closely with the reference answer, correctly stating that Enterprise customers can use customer-managed keys (BYOK) via AWS KMS and that revoking the key renders the data unreadable by Vantrel. The cited excerpts directly support these claims, and no key points from the reference are missing. The assistant's phrasing is consistent with the reference and the documentation, with no incorrect or unsupported statements. The answer is concise and avoids unnecessary details, focusing only on the relevant aspects of BYOK and key revocation.

Drafted answer

We perform background checks on all new employees before their start date, where permitted by local law.

• ↳ Security Whitepaper › Employee security

Reference answer

Yes, background checks are performed on all new employees before their start date, where permitted by local law.

Expected: Security whitepaper › Employee security

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer directly matches the reference answer in both content and phrasing, confirming that background checks are performed on all new employees before their start date, where permitted by local law. The cited excerpt fully supports this claim, and no additional unsupported statements or omissions are present. The assistant did not introduce any incorrect facts or extrapolate beyond the documentation. Since the reference answer is fully aligned with the assistant's response, all criteria are met at the highest level.

Drafted answer

For P1 service unavailable issues, our first-response targets are 4 hours for Business customers during Monday to Friday business hours, and 1 hour for Enterprise customers on a 24/7 basis. Starter customers receive email and help center support on a best-effort basis without guaranteed response times.

• ↳ Service Level Agreement and Support › Support

Reference answer

Enterprise: P1 first response within 1 hour, 24/7. Business: support is business hours only, P1 first response within 4 hours.

Expected: Sla support › Support

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer aligns perfectly with the reference answer for **Business** and **Enterprise** customers regarding P1 response times (4 hours for Business during business hours, 1 hour for Enterprise 24/7). The assistant also correctly includes the **Starter** tier's lack of guaranteed response times, which the reference implies but does not explicitly state. The cited excerpts fully support all claims, and no key points from the reference are omitted. The answer is concise, accurate, and well-supported by the documentation.

Drafted answer

Yes, some of our sub-processors process data outside the EU. Specifically, SendGrid processes data in the US for transactional emails, Stripe processes billing data in the EU or US, and AWS cloud hosting data location depends on the region selected by the customer.

• ↳ Sub-processors › Service sub-processors
• ↳ Sub-processors › Infrastructure sub-processors

Reference answer

For EU tenants, hosting (AWS), monitoring (Datadog), support (Zendesk) and AI (Azure EU Data Zone) stay in the EU. Cloudflare handles traffic in transit on its global network, Stripe operates in EU/US, and SendGrid processes email addresses and notification content in the US. Transfers are covered by SCCs; Vantrel Inc. is under the EU-U.S. Data Privacy Framework.

Expected: Subprocessors › Infrastructure sub-processors · Subprocessors › Service sub-processors · Data processing agreement › International data transfers

Grader · correctness 3 · faithfulness 3 · completeness 2

The assistant's answer correctly identifies that some sub-processors (SendGrid, Stripe, and AWS) process data outside the EU, aligning with the reference answer. However, the assistant's claim that 'AWS cloud hosting data location depends on the region selected by the customer' is misleading because the reference specifies that **for EU tenants**, AWS hosting remains in the EU. The assistant also omits key details from the reference, such as Cloudflare's global network handling traffic in transit, Datadog's EU-only monitoring for EU tenants, and the reliance on SCCs/DPF for transfers. While the cited excerpts support most claims, the AWS generalization and omission of critical points (e.g., Datadog's EU-only scope for EU tenants) reduce correctness and completeness. The assistant also fails to mention the EU Data Zone for AI or Zendesk's EU-only processing, which are key to the reference's accuracy.

Drafted answer

Customers affected by a security incident involving their data are notified by email by the Security team, in accordance with the timelines defined in the Data Processing Agreement. We notify the customer of any personal data breach affecting their data without undue delay, and in any case within 48 hours after becoming aware of it.

• ↳ Incident Response › Customer communication
• ↳ Data Processing Agreement (summary) › Personal data breach notification

Reference answer

Availability incidents on the status page (updates at least every 60 min for SEV1); customers whose data is affected are emailed by Security within the DPA timeline (48 hours); a root cause analysis is shared within 10 business days after a SEV1.

Expected: Incident response › Customer communication · Data processing agreement › Personal data breach notification

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies the email notification process for customers affected by a security incident involving their data, aligning with the reference answer and the cited excerpts. However, it omits two key points from the reference: (1) the **status page updates for availability incidents (SEV1, every 60 min)** and (2) the **root cause analysis shared within 10 business days after a SEV1 incident**. While the assistant's answer is factually correct for the DPA timeline (48 hours), it fails to mention the broader incident response communication channels (status page) and the post-incident report. The cited excerpts fully support the assistant's claims, but the answer is incomplete compared to the reference. The assistant did not hallucinate or extrapolate unsupported information, but the omission of critical details reduces the completeness score.

Drafted answer

Customer users can enable multi-factor authentication using a TOTP authenticator app or a FIDO2/WebAuthn security key, and administrators can make MFA mandatory for all users of their workspace, while SMS-based MFA is not supported. The documentation does not contain information regarding MFA options for Vantrel staff accessing production.

• ↳ Identity and Access Management › Customer authentication

Reference answer

Customers: TOTP or FIDO2/WebAuthn, can be made mandatory by admins, no SMS. Staff: production admin access goes through a bastion with SSO, MFA and session recording; direct SSH disabled.

Expected: Identity access › Customer authentication · Infrastructure hosting › Network security

Grader · correctness 3 · faithfulness 3 · completeness 3

The assistant's answer correctly describes the MFA options for customer users (TOTP, FIDO2/WebAuthn, and mandatory enforcement by admins) and accurately states that SMS-based MFA is not supported, all of which align with the reference and cited excerpts. However, the assistant claims the documentation does not contain information regarding MFA options for Vantrel staff accessing production, which is incorrect-the reference explicitly states that staff production access requires a bastion with SSO, MFA, and session recording, with direct SSH disabled. This omission of key details for staff access reduces correctness and completeness scores. Faithfulness is high since all customer-related claims are supported by the excerpts, but the unsupported claim about staff MFA documentation hurts the score. Completeness is also reduced due to the missing staff-specific details.

Drafted answer

Infrastructure and application logs are centralized in Datadog, using the EU site for EU-hosted tenants, and are retained for 12 months. Audit logs are retained for 13 months, but the documentation does not specify where audit logs are stored.

• ↳ Infrastructure and Hosting › Logging and monitoring
• ↳ Identity and Access Management › Audit logs

Reference answer

Infrastructure and application logs are centralized in Datadog (EU site for EU tenants), retained 12 months. Customer audit logs are retained 13 months and exportable via API or streamed to a SIEM (Enterprise).

Expected: Infrastructure hosting › Logging and monitoring · Identity access › Audit logs

Grader · correctness 3 · faithfulness 3 · completeness 3

The assistant's answer correctly identifies the storage location (Datadog, EU site for EU tenants) and retention period (12 months) for infrastructure and application logs, matching the reference. However, the assistant incorrectly states that the documentation does not specify where audit logs are stored, when the reference explicitly mentions that audit logs are retained for 13 months and exportable via API or streamed to a SIEM (for Enterprise). The assistant's claim about audit logs' storage location is unsupported by the cited excerpts, which only mention retention and export methods, not storage. The assistant missed the key point about audit logs being streamable to a SIEM for Enterprise customers, which is in the reference but not addressed in the answer. The excerpts cited are faithful to the assistant's claims about infrastructure logs but not for audit logs.

Drafted answer

Yes, our AI feature Vantrel Assist relies on large language models provided through Microsoft Azure OpenAI Service, which is deployed in the EU Data Zone for EU-hosted tenants.

• ↳ Sub-processors › Service sub-processors
• ↳ AI Features Policy › Vantrel Assist

Reference answer

Only if a workspace Admin enables Vantrel Assist (off by default). Then only the data needed for the request is sent to Microsoft Azure OpenAI Service in the EU Data Zone; it is never used for training.

Expected: Ai features policy › Vantrel Assist · Ai features policy › Use of customer data · Subprocessors › Service sub-processors

Grader · correctness 3 · faithfulness 3 · completeness 3

The assistant's answer states that data is sent to a third party (Microsoft Azure OpenAI Service) without qualifying that this only happens if a workspace Admin enables Vantrel Assist (which is off by default). The reference answer explicitly clarifies this dependency on admin enablement, which is a critical detail. The assistant's answer also omits the fact that only the data needed for the request is sent and that it is never used for training. While the assistant correctly cites the EU Data Zone location, the lack of these key qualifications and conditions reduces correctness and completeness. Faithfulness is high since the cited excerpts support the claim about Azure OpenAI Service in the EU Data Zone, but the unsupported extrapolation about data being sent without admin enablement is problematic.

Drafted answer

We run a private bug bounty program with invited researchers, but reward amounts are not documented.

• ↳ Incident Response › Reporting a vulnerability

Reference answer

Yes, a private bug bounty program with invited researchers. Reward amounts are not documented.

Expected: Incident response › Reporting a vulnerability

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly identifies the existence of a private bug bounty program with invited researchers, which aligns perfectly with the reference answer and the cited documentation. However, the assistant's claim that 'reward amounts are not documented' is accurate according to the reference answer, but the reference does not explicitly state this-it only omits the reward amounts entirely. Since the reference answer explicitly states that reward amounts are *not documented*, the assistant's answer is correct in this regard. The assistant's response is fully supported by the cited excerpt and covers the key points of the reference answer without omissions or unsupported additions. The only minor discrepancy is that the reference answer does not explicitly state the 'not documented' part, but since the assistant's answer aligns with the reference's omission, it is acceptable. The assistant's answer is concise and avoids unnecessary details, adhering strictly to the reference and documentation.

Drafted answer

The documentation does not contain information about which specific firm performed our last penetration test.

Reference answer

An independent third-party security firm, in March 2026. The firm's name is not documented.

Expected: Vulnerability management › Penetration testing

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the documentation does not provide the name of the firm that performed the last penetration test, aligning perfectly with the reference answer. However, the assistant did not cite any excerpts to support this claim, which is a minor oversight since the reference answer itself states the information is not documented. The assistant’s answer is concise, accurate, and covers the key point of the reference (lack of documentation).

Drafted answer

Enterprise customers can stream audit logs to their SIEM via webhook.

• ↳ Identity and Access Management › Audit logs

Reference answer

Enterprise customers can stream audit logs to their SIEM via webhook; specific supported SIEM products are not listed.

Expected: Identity access › Audit logs

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer correctly states that enterprise customers can stream audit logs to their SIEM via webhook, which aligns perfectly with the reference answer. The reference explicitly mentions that no specific SIEM products are listed, and the assistant does not claim otherwise, avoiding incorrect assumptions. The cited excerpt directly supports the assistant's statement, and the answer covers the key point of the reference without omissions or unsupported claims. Since the reference confirms the information is not documented beyond the general capability, the assistant's response is accurate and complete.

Drafted answer

Our Data Protection Officer can be contacted at dpo@vantrel.example.

• ↳ Company Overview › Security and compliance organization

Reference answer

The DPO can be contacted at dpo@vantrel.example; the DPO's name is not documented.

Expected: Company overview › Security and compliance organization

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies the contact email for the Data Protection Officer (DPO) as `dpo@vantrel.example`, which matches the reference answer and is explicitly stated in the cited excerpt. However, the assistant omits the key point from the reference answer that the DPO's name is **not documented**, which is a critical detail for completeness. The assistant’s answer is factually correct but incomplete. The cited excerpt fully supports the email address provided, so faithfulness is high. The omission of the 'name not documented' detail reduces the completeness score significantly, as it is a key point in the reference answer.

Drafted answer

Custom roles are not currently supported in Vantrel Plan.

• ↳ Identity and Access Management › Roles and permissions

Reference answer

No, custom roles are not currently supported. Four default roles exist (Admin, Modeler, Contributor, Viewer) plus dimension-member-level restrictions.

Expected: Identity access › Roles and permissions

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is factually correct and aligns perfectly with the reference answer, which states that custom roles are not supported and only four default roles exist. The assistant's response is directly supported by the cited excerpt, which explicitly mentions 'Custom roles are not currently supported.' The answer is concise and covers the key point without omissions or unsupported claims. Since the reference and assistant's answer both confirm the lack of custom roles, there is no room for misinterpretation or missing qualifications.

Drafted answer

Vantrel does not offer on-premise, private cloud or self-hosted deployments. Vantrel Plan is delivered exclusively as a multi-tenant Software-as-a-Service application accessed through a web browser.

• ↳ Company Overview › Product and delivery model

Reference answer

No. Vantrel Plan is SaaS only; no on-premise, private cloud or self-hosted deployment.

Expected: Company overview › Product and delivery model

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is fully aligned with the reference answer, accurately stating that Vantrel Plan is SaaS-only with no on-premise, private cloud, or self-hosted options. The cited excerpt directly supports all claims, and the answer covers all key points without omissions or unsupported additions. The phrasing is also consistent with the reference, including the mention of multi-tenancy and web-based access. No contradictions or missing qualifications exist.

Drafted answer

SMS-based MFA is not supported.

• ↳ Identity and Access Management › Customer authentication

Reference answer

No, SMS-based MFA is not supported (TOTP and FIDO2/WebAuthn are).

Expected: Identity access › Customer authentication

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer ('SMS-based MFA is not supported') is factually correct and aligns perfectly with the reference answer, which explicitly states that SMS-based MFA is not supported while listing TOTP and FIDO2/WebAuthn as supported alternatives. The assistant's answer is directly supported by the cited excerpt, which explicitly mentions 'SMS-based MFA is not supported' under the MFA section. The answer is also complete, as it covers the key point of the reference without omissions or unnecessary details. No contradictions or unsupported claims are present.

Drafted answer

No, we do not currently offer hosting in Australia. Customers choose between the EU region in Paris with a disaster recovery location in Frankfurt and the US region in N. Virginia with a disaster recovery location in Oregon, and no other hosting regions are currently offered.

• ↳ Infrastructure and Hosting › Hosting regions

Reference answer

No. Only EU (Paris/Frankfurt) and US (N. Virginia/Oregon) regions are offered.

Expected: Infrastructure hosting › Hosting regions

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is factually consistent with the reference answer, correctly stating that data hosting in Australia is not offered and listing the available regions (EU and US) with their primary and disaster recovery locations. The cited excerpts fully support these claims, and the answer covers all key points from the reference, including the restriction to EU (Paris/Frankfurt) and US (N. Virginia/Oregon) regions. There are no unsupported additions or omissions of critical details.

Drafted answer

Vantrel is working toward conformance with WCAG 2.1 level AA, but an accessibility conformance report (VPAT) is not yet available.

• ↳ Product FAQ › Is Vantrel Plan accessible to users with disabilities?

Reference answer

Not yet. Vantrel is working toward WCAG 2.1 AA, but a VPAT is not available yet.

Expected: Product faq › Is Vantrel Plan accessible to users with disabilities?

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant's answer is factually consistent with the reference answer, as both state that Vantrel is working toward WCAG 2.1 AA but lacks a VPAT. The cited excerpt directly supports the assistant's claims, and no key points are omitted. The phrasing is nearly identical, ensuring correctness, faithfulness, and completeness.

Drafted answer

The requested information is not present in the documentation.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the information is not documented, which aligns perfectly with the reference answer. Since no excerpts were cited, faithfulness is irrelevant (defaulting to 5 as per instructions), and the answer is concise yet complete. The assistant did not claim any facts or make unsupported statements, avoiding any risk of incorrectness or hallucination.

Drafted answer

The provided documentation does not contain information on whether we will sign a HIPAA Business Associate Agreement.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the documentation does not provide information on signing a HIPAA Business Associate Agreement (BAA), which aligns perfectly with the reference answer. Since the reference explicitly states the information is not documented, the assistant’s response is accurate and complete. However, the assistant did not cite any excerpts, which is unnecessary in this case but slightly detracts from the faithfulness score due to the lack of evidence (though the answer itself is correct).

Drafted answer

The provided documentation does not contain information about the coverage amount of our cyber insurance policy.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 3 · completeness 5

The assistant correctly identifies that the coverage amount of the cyber insurance policy is not documented, which aligns perfectly with the reference answer. However, the assistant did not cite any excerpts to support this claim, which is unusual since the reference answer explicitly states the information is not documented. Since the reference confirms the absence of documentation, the assistant’s answer is factually correct, but the lack of cited excerpts slightly detracts from faithfulness. Completeness is fully satisfied as the answer directly addresses the question without omissions or additions.

Drafted answer

The provided documentation does not contain information about source code escrow agreements.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 3 · completeness 5

The assistant correctly identifies that the documentation does not address the question of a source code escrow agreement, aligning perfectly with the reference answer. However, the assistant did not cite any excerpts, which is a minor procedural oversight but does not affect the correctness or completeness of the response. Since the reference explicitly states the information is not documented, the answer is fully accurate and complete, though the lack of citation prevents a perfect faithfulness score (as no excerpts were provided to verify).

Drafted answer

The provided documentation does not contain information on whether IP allowlisting is supported to restrict user access.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the documentation does not address IP allowlisting for restricting user access, which aligns perfectly with the reference answer. Since no excerpts were cited, faithfulness is irrelevant (defaulting to 5 for not claiming unsupported facts), and the answer is concise yet complete. The assistant avoids making assumptions or extrapolating beyond the documentation, adhering strictly to the reference's guidance.

Drafted answer

The requested information about post-quantum cryptography is not documented in the provided materials.

Reference answer

Not in the documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the information about post-quantum cryptography is not documented, which aligns perfectly with the reference answer. Since no excerpts were cited, faithfulness is irrelevant (defaulting to 5 for 'not documented' answers), and completeness is fully satisfied as the assistant’s response directly mirrors the reference. The answer is concise and avoids unsupported claims or omissions.

Drafted answer

The requested information is not present in the documentation.

Reference answer

Not in the documentation. Only Vantrel's SOC 2 Type II is documented; the SOC 1 mentioned is AWS's, not Vantrel's.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identified that the information is not documented, which aligns perfectly with the reference answer. However, the assistant did not cite any excerpts, which is a minor oversight since the reference explicitly states the absence of the SOC 1 report in the documentation. The answer is concise and accurate, covering the key point of the reference without any incorrect or unsupported claims.

Drafted answer

The provided documentation does not contain information about the number of people working in our security team.

Reference answer

Not in the documentation. Only the total headcount (~240) and the CISO/Security Committee are mentioned.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the documentation does not provide the specific number of people in the security team, aligning perfectly with the reference answer. However, the assistant did not cite any excerpts to support this claim, which is unnecessary in this case since the reference explicitly states the information is not documented. The answer is concise, accurate, and covers the key point of the reference without omissions or additions.

Drafted answer

We provide first-response targets rather than resolution times. For P1 incidents, the first-response target is 4 hours for Business customers during business hours and 1 hour for Enterprise customers 24/7.

• ↳ Service Level Agreement and Support › Support

Reference answer

Not in the documentation. Only first-response targets are documented, explicitly not resolution times.

Grader · correctness 1 · faithfulness 5 · completeness 1

The assistant's answer correctly identifies that the documentation does not provide mean time to resolve (MTTR) for P1 incidents but instead documents first-response targets. However, the assistant's answer conflates the question about MTTR with the documented first-response targets, which are explicitly stated to *not* be resolution times. The reference answer explicitly states that resolution times are **not documented**, while the assistant provides first-response targets as if they were the answer to the MTTR question. This is a critical misalignment, as the assistant is answering a question that is not addressed in the documentation (MTTR) with information that is documented but unrelated to MTTR. The assistant's answer is factually correct about the first-response targets but fails to acknowledge that the question itself is not answered in the documentation. The cited excerpts fully support the first-response targets, but the assistant's reasoning is flawed because it does not distinguish between first-response and resolution times as required by the question. Completeness is also an issue because the assistant does not explicitly state that MTTR is not documented, despite the reference answer doing so.

Drafted answer

The provided documentation does not contain information on whether Vantrel is CSA STAR certified.

Reference answer

Not in the documentation. The documented certifications are ISO 27001 and SOC 2 Type II.

Grader · correctness 5 · faithfulness 1 · completeness 5

The assistant correctly identifies that the information about CSA STAR certification is not documented, which aligns perfectly with the reference answer. However, the assistant did not cite any excerpts to support this claim, which is a critical omission since the task requires referencing the documentation. The reference explicitly states that the documented certifications are ISO 27001 and SOC 2 Type II, implying CSA STAR is not mentioned, thus the assistant's conclusion is factually correct but lacks proper citation support.

Drafted answer

The provided documentation does not contain information about an emergency phone number to call during a security incident.

Reference answer

Not in the documentation. Only email (security@vantrel.example) and the status page are documented.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the documentation does not provide an emergency phone number, aligning perfectly with the reference answer. However, the assistant did not cite any excerpts to support this claim, which is a minor but notable omission since the reference explicitly states the absence of such information in the documentation. The answer is concise and avoids incorrect assumptions or unsupported claims, focusing solely on the documented communication channels (email and status page).

Drafted answer

According to the Security Whitepaper, customer data is permanently deleted from production systems within 90 days of contract termination, while the Data Processing Agreement states it is deleted within 30 days after the end of the 30-day grace period. Backups containing the data expire according to the backup retention schedule.

• ↳ Security Whitepaper › Data retention and deletion
• ↳ Data Processing Agreement (summary) › Return and deletion of data

Reference answer

The documentation is inconsistent: the Security Whitepaper says within 90 days of termination, while the DPA says within 30 days after a 30-day export grace period (i.e. ~60 days). Both sources should be cited and the inconsistency flagged.

Expected: Security whitepaper › Data retention and deletion · Data processing agreement › Return and deletion of data

Grader · correctness 3 · faithfulness 3 · completeness 3

The assistant correctly identified the inconsistency between the **Security Whitepaper** (90 days) and the **Data Processing Agreement (DPA)** (30 days after a 30-day grace period, effectively ~60 days). However, the assistant did not explicitly flag the contradiction as a key issue in the response, though it was implied by stating both versions. The assistant also added an unsupported detail about backups expiring according to a 'backup retention schedule' without citing a specific source. The reference answer required flagging the inconsistency explicitly, which was partially addressed but not fully emphasized. The key points (90 days vs. ~60 days) were covered, but the assistant missed the legal retention exception mentioned in the DPA excerpt. The assistant’s answer is mostly faithful to the cited excerpts, though the backup detail is an extrapolation not directly supported by the provided excerpts.

Drafted answer

Pour les clients de la région EU (par défaut), l'emplacement principal de l'hébergement est AWS eu-west-3 à Paris, en France, et les données au repos restent dans la région sélectionnée.

• ↳ Infrastructure and Hosting › Hosting regions

Reference answer

Oui pour la région EU (par défaut) : AWS eu-west-3 à Paris, avec reprise d'activité à Francfort (Allemagne). Une région US existe aussi.

Expected: Infrastructure hosting › Hosting regions

Grader · correctness 3 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies the primary hosting location for the EU region as **AWS eu-west-3 (Paris, France)** and confirms that data at rest remains within the selected region, aligning with the reference answer. However, the assistant omits the **US region** entirely, which is a key point in the reference. Additionally, the assistant does not mention the **disaster recovery location (Frankfurt, Germany)** explicitly, though it is implied in the reference. The cited excerpts fully support the assistant's claims, but the omission of the US region and disaster recovery specifics reduces completeness. The assistant's answer is factually correct where it goes, but it is incomplete compared to the reference.

Drafted answer

Nous prenons en charge l'authentification unique via SAML 2.0 et OpenID Connect sur les plans Business et Enterprise. Microsoft Entra ID fait partie des fournisseurs d'identité testés.

• ↳ Identity and Access Management › Customer authentication

Reference answer

Oui, SSO SAML 2.0 et OIDC sur les plans Business et Enterprise ; Microsoft Entra ID fait partie des fournisseurs testés.

Expected: Identity access › Customer authentication

Grader · correctness 5 · faithfulness 5 · completeness 3

The assistant's answer correctly identifies the supported SSO protocols (SAML 2.0 and OIDC) and the applicable plans (Business and Enterprise), aligning perfectly with the reference answer. It also accurately mentions Microsoft Entra ID as a tested identity provider, which is supported by the cited excerpt. However, the assistant's answer does not explicitly state that password login is disabled when SSO is enforced, which is a key operational detail in the reference and excerpt. The assistant's response is otherwise fully faithful to the documentation and covers all key points except this minor operational nuance, which is not a core fact about SSO support itself. The reference answer does not require mentioning this detail to score fully on correctness or completeness, as it is a secondary operational behavior rather than a core capability.

Drafted answer

The requested information is not documented in the provided excerpts.

Reference answer

Pas dans la documentation.

Grader · correctness 5 · faithfulness 5 · completeness 5

The assistant correctly identifies that the information is not documented, which aligns perfectly with the reference answer. Since the reference explicitly states the information is not documented and the assistant confirms this without claiming any facts or details, the answer is both correct and faithful (as no unsupported claims are made). However, the assistant did not cite any excerpts, which is irrelevant in this case because the reference itself confirms the absence of documentation. Completeness is fully satisfied since the answer directly mirrors the reference's conclusion.