Knowledge Hub · Questions & Challenges

The questions people actually ask us.

Practical questions — cost, timelines, "do we need this" — plus the common challenges teams run into during implementation, across every practice. For term definitions, see the Glossary; for step-by-step requirements, see each practice's checklists.

Cybersecurity & Cyber Assurance

How much does a penetration test typically cost?

Cost varies by scope (a single web app vs. a full network vs. a whole organization) and depth (black-box vs. white-box) — from a focused few-day engagement to a multi-week assessment for larger environments. Get a scoped quote rather than relying on a generic number.

Do we need a VAPT if we already run automated vulnerability scans?

Yes. Automated scanning catches known, signature-based issues; a VAPT adds human-led, context-aware exploitation attempts — including business-logic flaws — that scanners can't replicate.

How often should we run a penetration test?

At least annually for most organizations, plus after any significant infrastructure or application change, and increasingly paired with continuous, lighter-touch testing in between.

Is cyber insurance a substitute for security controls?

No. Insurers increasingly require evidence of baseline controls — MFA, patching, backups, incident response — before binding a policy or paying out a claim, so insurance supplements a security programme rather than replacing one.

Do small businesses really need to worry about non-human identity risk?

Yes. Even a small environment typically runs more service accounts, API keys and automation scripts than human users, and these are often the least monitored, most over-privileged accounts in the environment.

What's a reasonable first step if we've never had a security assessment?

Start with a lightweight gap assessment against a recognized framework, such as CIS Controls or NIST CSF, to find the highest-risk gaps — rather than jumping straight to a full VAPT or certification effort.

What's the difference between a penetration test and a red team exercise?

A penetration test targets a defined scope — an app, a network segment — to find and demonstrate as many exploitable flaws as practical within a set timeframe. A red team exercise simulates a realistic adversary pursuing a specific objective (like reaching sensitive data) stealthily, testing detection and response as much as the technical defenses themselves.

Do we have to remediate every finding, even low-severity ones?

Not immediately, but every finding needs a documented decision — fix, accept the risk, or compensate with another control. Unaddressed low-severity findings are a common audit red flag precisely because "we'll get to it" so often means "we never did."

Is a web application firewall (WAF) enough to protect a public-facing app?

No. A WAF blocks known attack patterns at the network edge, but it doesn't fix the underlying vulnerable code, and sophisticated or novel attacks can bypass it entirely. It's a useful compensating control, not a substitute for secure development and testing.

How should we scope a penetration test — black-box, grey-box or white-box?

Grey-box (testers get some internal context, like a standard user account) usually gives the best balance of realism and depth for most organizations. Pure black-box is more realistic but can burn engagement time on reconnaissance; white-box (full source/architecture access) is best for deep assessments of a specific high-risk system.

What's a reasonable SLA for patching a critical vulnerability?

Most mature programmes target critical, actively-exploited vulnerabilities within 24–72 hours and other critical findings within 2 weeks, with longer windows for high/medium severity — but the number matters less than having a documented SLA at all and tracking adherence to it.

Do we need our own SOC, or is a managed detection and response (MDR) service enough?

Most small and mid-size organizations are better served by an MDR/SOC-as-a-service provider — 24/7 in-house monitoring is expensive to staff well. Build an internal SOC only once scale and risk genuinely justify the dedicated headcount.

Common challenges & how to avoid them

Security treated as a one-time project, not an ongoing programme

A single VAPT or audit creates a false sense of "done," when real risk keeps changing as systems, staff and threats evolve.

How to avoid itBudget for continuous testing, monitoring and periodic reassessment, not a single point-in-time event.

Over-privileged service accounts going unnoticed

Non-human identities multiply faster than anyone tracks them, and each one often gets broad access "to be safe."

How to avoid itRun a periodic access review specifically for service accounts and API keys, not just human users.

Alert fatigue drowning out real incidents

Poorly tuned SIEM and monitoring tools generate so many low-value alerts that real threats get missed in the noise.

How to avoid itInvest in tuning and prioritization before adding more detection tools on top.

No tested incident response plan

Many organizations have a written IR plan that's never been rehearsed, so the first real test happens during an actual breach.

How to avoid itRun at least one tabletop exercise a year, and update the plan based on what it reveals.

No real asset inventory, so parts of the attack surface stay invisible

Shadow IT, forgotten subdomains and unmanaged cloud resources accumulate outside any security review — you can't protect what you don't know exists.

How to avoid itMaintain a living asset inventory fed by automated discovery, not a spreadsheet updated once a year.

Patch management stalls for fear of breaking production

Teams delay applying fixes because a past patch once caused an outage, so "stability" quietly wins over "secure."

How to avoid itPatch in a staged environment first with a fast rollback plan, so speed and stability stop being a trade-off.

MFA treated as the finish line instead of one layer

Push-bombing and MFA-fatigue attacks exploit the fact that many teams stop hardening identity once MFA is switched on.

How to avoid itPair MFA with phishing-resistant methods and anomaly-based login monitoring, not a single static factor.

Findings get reported but remediation is never actually tracked to closure

A report full of findings creates the appearance of progress, but without a tracked remediation backlog, the same issues reappear in next year's test.

How to avoid itFeed every finding into a tracked backlog with an owner and due date, and re-test against it, not just against the original scope.

ISO & Compliance Implementation

How much does ISO 27001 certification cost?

Costs span three buckets: internal implementation effort, any external consulting support, and the certification body's audit fees. They scale with organization size and scope — most mid-size organizations should budget several months of dedicated effort alongside the direct audit fees.

Can we get certified without hiring a consultant?

Yes, if you already have an experienced internal owner and reasonable security maturity. Many organizations do succeed alone, but most find a consultant pays for itself by avoiding costly mistakes in scoping and the Statement of Applicability.

What happens if we fail the certification audit?

A major nonconformity pauses certification until it's corrected and verified, usually requiring a follow-up audit. A minor nonconformity typically just needs a documented corrective action plan before certification is issued.

Do we need to recertify every year?

No. ISO certification is valid for three years, with shorter annual or 18-month surveillance audits in between to confirm the management system is still being maintained.

Can one ISMS cover multiple certifications, like ISO 27001 and ISO 27701 together?

Yes. ISO 27701 is explicitly designed as an extension to an existing ISO 27001 ISMS, though it can now also be certified standalone — organizations commonly run them as one integrated management system rather than two parallel efforts.

Should we pursue SOC 2 or ISO 27001 first, or do we eventually need both?

Many organizations start with whichever their most important customers or market demands first, then add the other as their customer base diversifies. Holding both isn't unusual for companies selling into both US and international enterprise markets.

What's the difference between ISO 27001:2013 and the 2022 revision?

The 2022 revision restructured Annex A from 114 controls across 14 domains into 93 controls across 4 themed groups, and added new controls reflecting modern risk — cloud security, threat intelligence, data masking and secure coding among them. Organizations certified under the old version have a transition deadline to migrate.

What happens in a Stage 1 vs. Stage 2 audit?

Stage 1 is a documentation review — the auditor checks your ISMS scope, policies and Statement of Applicability are complete and ready for assessment. Stage 2 is the real test: the auditor checks for actual operating evidence that the controls you documented are genuinely being followed.

Can a startup with under 20 people realistically get ISO 27001 certified?

Yes — company size doesn't determine certifiability, scope and risk do. Small teams often move faster precisely because there's less legacy process to untangle, though they do need to resource the effort seriously rather than treat it as a side project.

What's the difference between a nonconformity and an observation in an audit?

A nonconformity means a control isn't operating as required — major ones block certification until fixed and verified, minor ones need a corrective action plan. An observation flags something that isn't a current failure but could become one, and doesn't block certification, though ignoring it repeatedly often turns it into a future nonconformity.

Does getting ISO 27001 certified also make us DPDPA or GDPR compliant?

No, but it helps substantially. ISO 27001 builds the security backbone — risk management, access control, incident response — that privacy regulations assume exists, but DPDPA/GDPR add their own distinct requirements around consent, data subject rights and lawful basis that ISO 27001 doesn't cover on its own.

Is SOC 2 Type I sufficient, or do we need Type II?

Type I confirms your controls are designed correctly at a single point in time; Type II confirms they actually operated effectively over a review period, typically 6–12 months. Most enterprise buyers specifically ask for Type II because it's real evidence of sustained practice, not just a documented design.

Common challenges & how to avoid them

Treating the Statement of Applicability as a formality

Teams mark most controls "applicable" without real justification, which an auditor will immediately probe.

How to avoid itTie every applicability decision directly back to the risk assessment, not convenience.

Policies that exist on paper but aren't actually followed

A well-written policy with no evidence of real operational practice is one of the most common audit findings.

How to avoid itBuild evidence-collection into daily operations, not just document creation right before the audit.

Underestimating the time to embed a genuine security culture

Certification can be achieved faster than real behavior change spreads across the organization.

How to avoid itStart staff training and awareness early, in parallel with technical control implementation — not after.

Audit fatigue from running disconnected compliance efforts

Separate teams chasing ISO 27001, SOC 2 and sector-specific requirements independently duplicate evidence-gathering work.

How to avoid itMap overlapping controls across frameworks once, and reuse evidence across audits — a unified control framework or GRC platform helps.

Risk assessment done once at kickoff and never revisited

The risk register that drove initial control selection quietly goes stale as the business adds new systems, vendors and products.

How to avoid itReview and update the risk assessment on a defined cadence — and whenever a significant change happens, not just annually by default.

Internal audit treated as a formality right before the external one

A rushed, superficial internal audit misses the same gaps the external auditor will find — at a much worse time to discover them.

How to avoid itRun the internal audit early enough, and critically enough, that its findings are actually fixed before the real audit.

Evidence scattered across emails, drives and individual inboxes

Without a central repository, gathering audit evidence becomes a last-minute scramble that eats weeks of everyone's time.

How to avoid itCentralize evidence collection in a GRC tool or shared structure from day one, tied to each control.

New hires never actually completing security awareness training

Training gets assigned on paper during onboarding but completion is rarely tracked, so it quietly doesn't happen.

How to avoid itTrack training completion as a hard gate, not a suggestion, and re-certify annually.

Privacy & Data Protection

Does DPDPA apply to us if we're based outside India?

Yes, if you offer goods or services to individuals in India and process their digital personal data in that context. DPDPA has extraterritorial reach similar to GDPR's, though specifics should be confirmed with counsel for your situation.

What's the penalty for DPDPA non-compliance?

DPDPA sets financial penalties that scale with the nature and severity of the breach, decided by the Data Protection Board case by case — there isn't a single fixed number, which is part of why documented, demonstrable compliance effort matters even before any incident occurs.

Do we need both a DPO and a Consent Manager?

Not necessarily both. A DPO is only mandatory if you're designated a Significant Data Fiduciary; a Consent Manager is a separate, optional entity individuals can choose to manage their consents through, not something every organization needs to appoint itself.

How long does it take to become DPDPA-ready from scratch?

Timelines vary widely based on how much of a data inventory, consent infrastructure and security baseline already exists. Organizations starting from nothing should budget several months for a genuine first pass, not weeks.

Is anonymized data still subject to DPDPA?

No. Once data is genuinely and irreversibly anonymized so it can never be linked back to an individual, it falls outside DPDPA's scope. Pseudonymized data, which can still be re-identified, remains regulated.

What should we do first if we've never done any privacy work?

Start with a data mapping exercise — you can't assess risk, write accurate notices, or respond to a rights request for data you haven't inventoried.

Do we need separate consent for every single processing purpose?

Generally yes — DPDPA requires consent to be specific and granular, not one broad blanket acceptance covering unrelated purposes. Bundling marketing consent with, say, account-creation consent is a common compliance gap.

When do we need to do a Data Protection Impact Assessment (DPIA)?

Before any processing activity that's likely to pose a significant risk to individuals — large-scale sensitive data processing, new tracking technology, profiling with legal or similar effects. Doing it before launch, not after, is what makes it useful.

Does DPDPA restrict transferring data outside India?

DPDPA takes a more permissive approach than GDPR — cross-border transfer is generally allowed except to countries the government specifically restricts, rather than requiring a transfer mechanism for every destination. Confirm current restricted-country status with counsel, since this list can change.

Does DPDPA cover our own employees' data, or only customers'?

It covers employees too, with a narrow carve-out for processing necessary for employment purposes like safety and HR administration. Marketing-style processing of employee data still needs a proper lawful basis.

How is DPDPA different from GDPR in day-to-day practice?

DPDPA is lighter on prescriptive mechanism (no formal "legitimate interest" balancing test, fewer mandated documents) but sets potentially very large penalties and gives individuals clear rights to access, correct and erase data — the practical compliance work of mapping data and honoring rights requests looks similar either way.

What's the breach notification timeline under DPDPA?

Organizations must notify the Data Protection Board and affected individuals, though exact timelines are still being finalized in implementing rules — the safe practice is to treat it like GDPR's 72-hour standard until the rules confirm otherwise, since being ready for the stricter timeline costs little extra.

Common challenges & how to avoid them

Consent language that's technically compliant but practically meaningless

Long, legalistic notices get agreed to without being read, which regulators increasingly scrutinize.

How to avoid itWrite notices in plain language, and test that a non-lawyer can actually understand what they're agreeing to.

No one actually owns the data inventory after it's built

Data mapping is often a one-time project exercise that goes stale the moment a new system or vendor is added.

How to avoid itAssign an ongoing owner, and trigger a data-map update whenever a new processing activity starts.

Rights requests falling through the cracks

Without a documented process, an access or deletion request can land in a generic inbox and simply get missed.

How to avoid itBuild a defined intake channel, owner and SLA for every category of data-principal request.

Vendor contracts that don't reflect actual data flows

A signed DPA often doesn't get revisited as a vendor relationship evolves and starts touching more data than originally scoped.

How to avoid itReview vendor data-processing scope against the contract periodically, not just at onboarding.

Cookie banners that don't actually block trackers until consent is given

Analytics and ad-tech scripts often fire on page load regardless of what the banner says, which is itself a compliance violation, not just a bad look.

How to avoid itTechnically gate non-essential scripts behind the consent decision, and verify it with a real browser test, not just the banner's presence.

A retention policy that exists on paper but deletion never actually happens

Data quietly accumulates past its stated retention period because no automated process actually enforces the deletion schedule.

How to avoid itAutomate deletion against the documented retention schedule rather than relying on someone remembering to run it.

Marketing collecting or using data outside the documented lawful basis

Growth teams adopt new tracking or enrichment tools faster than privacy review can keep up, quietly expanding processing beyond what was ever disclosed.

How to avoid itRequire a quick privacy check before any new data tool or tracking pixel goes live, not after it's already collecting.

Breach response plan that's never been tested

A plan that looks complete on paper often reveals missing steps, unclear ownership or unrealistic timelines the moment it's actually rehearsed.

How to avoid itRun a tabletop breach-notification drill at least annually, timing how long each step genuinely takes.

Accessibility & Inclusive Product Engineering

Is WCAG legally required, or just a best practice?

It depends on jurisdiction and sector. WCAG itself is a technical standard, not a law, but it's referenced by binding regulations like the European Accessibility Act and Section 508, and increasingly by ADA case law — so treating it as optional carries real legal risk in many markets.

How much does an accessibility audit cost?

Cost scales with product size and testing depth — a lightweight automated-plus-spot-check review costs far less than a comprehensive manual audit with real assistive-technology and screen-reader testing across every key flow.

Can we just buy an overlay widget instead of fixing our code?

No. Overlays are widely found not to achieve real WCAG conformance, since they can't fix underlying markup, keyboard operability or structural issues, and several have themselves faced accessibility-related lawsuits.

Do we need to test with real users with disabilities, or is automated testing enough?

Automated tools catch roughly a third of WCAG issues at most. Manual testing, and ideally real assistive-technology users, are needed to catch the rest, especially around usability and cognitive accessibility.

What's the fastest way to reduce our legal exposure if we're starting from zero?

Fix the highest-impact, most commonly litigated issues first — missing alt text, keyboard traps, insufficient color contrast — while scoping a full audit, rather than waiting for a comprehensive fix before addressing anything.

Does accessibility work slow down our development velocity?

It adds the most friction when retrofitted late. Built into design systems and component libraries from the start, accessible patterns become the default and cost little to no extra time per feature.

What conformance level should we target — A, AA or AAA?

AA is the standard target referenced by almost every regulation and procurement requirement worldwide, including the European Accessibility Act. AAA is rarely required in full and some of its criteria are genuinely impractical for certain content types — aim for AA, with select AAA criteria where they're a natural fit.

Does accessibility only apply to government and public-sector websites?

No. The European Accessibility Act extends requirements to private-sector e-commerce, banking and other consumer-facing digital services, and ADA litigation in the US has overwhelmingly targeted private companies, not government sites.

Do native mobile apps need to meet WCAG too?

Yes, in substance. WCAG was written with web content in mind, but its principles map directly onto mobile platform accessibility guidelines (Apple's and Android's), and VPATs and audits routinely cover native apps against the same success criteria.

What's the difference between a VPAT and an accessibility audit?

An audit is the testing process that finds and documents actual conformance issues. A VPAT (Voluntary Product Accessibility Template) is a standardized document — often requested by enterprise and government procurement — that reports conformance status per criterion, ideally backed by that audit's real findings.

Can AI-generated alt text replace a human writing it?

For simple, literal images it's a reasonable starting draft, but AI alt text regularly misses context that matters — what the image means in that specific page's purpose — and can be confidently wrong. Treat it as a first pass a human reviews, not a final answer.

Why do accessibility overlay widgets keep getting sued despite being marketed as a fix?

Overlays operate client-side and can't repair the underlying HTML, keyboard traps or missing semantic structure that cause most real accessibility failures — and several overlay vendors have themselves been named in accessibility lawsuits for exactly that gap between marketing and reality.

Common challenges & how to avoid them

Accessibility treated as a launch-week checklist, not a design discipline

Teams bolt on fixes right before release instead of designing accessibly from the start, which is slower and catches less.

How to avoid itBuild accessible patterns into your design system and component library so every new feature inherits them.

Automated scan results mistaken for full conformance

A clean automated report only reflects what automation can check — roughly a third of WCAG criteria.

How to avoid itPair automated scanning with manual and assistive-technology testing before calling anything conformant.

Accessibility backlog never gets prioritized against feature work

Without a clear owner or policy, accessibility fixes consistently lose to new feature requests.

How to avoid itSet a conformance target and track it as a release gate, not an optional backlog item.

VPAT filled in without evidence behind it

Procurement teams increasingly verify claims, and an inaccurate VPAT damages trust and can create legal exposure.

How to avoid itBack every claimed success criterion with an actual test result, not an assumption.

PDFs and documents left unremediated after the website itself is fixed

Teams fix the site and consider accessibility "done," while linked PDFs, forms and reports — often just as legally in-scope — stay completely inaccessible.

How to avoid itInclude downloadable documents in the audit scope and remediation plan from the start, not as a separate afterthought.

Testing done only with a mouse, never with a keyboard or screen reader

Visual QA alone misses keyboard traps, missing focus indicators and broken screen-reader announcements — the exact failures that block real assistive-technology users.

How to avoid itMake keyboard-only and screen-reader passes a standard, required part of QA, not an optional extra.

Third-party embedded widgets breaking an otherwise-accessible page

A chat widget, payment form or embedded video from a vendor can reintroduce keyboard traps or contrast failures the rest of the page already fixed.

How to avoid itAccessibility-test third-party embeds before integrating them, and re-test after any vendor update.

GenAI & Agentic AI Development

Do we need ISO 42001 certification to deploy AI responsibly?

Not strictly required by law in most jurisdictions today, but it's becoming a meaningful trust signal for enterprise customers and procurement teams — similar to the role ISO 27001 plays for security.

How much autonomy should an AI agent have in production?

Start narrow — scope autonomous action to low-consequence, easily reversible tasks, and require human approval for anything high-consequence or hard to reverse, expanding autonomy only as the agent proves reliable.

Is prompt injection a real risk, or mostly theoretical?

It's a real, demonstrated risk, especially for agents that retrieve content from the web, documents or tool outputs — any untrusted content an agent reads is a potential injection vector, not just the user's direct prompt.

Do open standards like MCP reduce our AI security risk?

They reduce integration risk and vendor lock-in by standardizing how agents connect to tools, but they don't eliminate the need for your own guardrails, permissioning and monitoring around what an agent is allowed to do with that access.

How do we decide between fine-tuning and RAG for our use case?

RAG is usually the faster, cheaper starting point for grounding a model in your own data and is easier to keep current. Fine-tuning suits teaching a model a specific style, format or specialized skill that retrieval alone can't achieve.

Does the EU AI Act apply to us if we're not in the EU?

Potentially yes — like GDPR, the EU AI Act has extraterritorial reach for AI systems whose output is used within the EU, regardless of where the provider is based.

Can hallucination ever be fully eliminated?

No, not with current generative architectures — it can be substantially reduced through grounding (RAG), guardrails and verification layers, but treat "zero hallucination" as a marketing claim to be skeptical of, not an achievable engineering target.

What's actually different between an AI agent and a chatbot?

A chatbot responds to messages within a conversation. An agent additionally plans, calls tools or APIs, and takes multi-step actions toward a goal with limited supervision — the distinction that matters for risk is autonomy and the ability to act on the world, not just converse about it.

Should we build on a commercial foundation model API or self-host an open-weight model?

APIs are faster to start with and offload infrastructure and model-upkeep burden; self-hosting an open-weight model gives more control over data residency, cost at scale and customization. Most organizations start with an API and only self-host once a specific requirement — cost at volume, strict data residency — justifies the added operational burden.

What's "shadow AI" and why does it matter?

Employees using unapproved consumer AI tools — pasting customer data into a public chatbot, for instance — outside any governance or security review. It's increasingly one of the most common sources of unintentional data exposure, precisely because it happens informally and goes unmonitored.

Do we need to disclose when content is AI-generated?

Increasingly yes, depending on jurisdiction and context — several emerging regulations (including provisions of the EU AI Act) require disclosure for certain AI-generated content, and customer trust considerations argue for transparency even where it isn't yet strictly mandated.

How much does unmonitored AI API usage typically cost organizations?

Enough that it's become its own category of financial risk — token costs scale with usage in ways that are easy to lose track of, especially with agents that make multiple model calls per task or get stuck in retry loops.

Common challenges & how to avoid them

Agents given broad tool access "to be safe" instead of least privilege

It's faster to grant wide permissions upfront than to scope them carefully, but this turns a single prompt-injection success into a major incident.

How to avoid itScope every tool and permission to exactly what the agent's task requires, and review it as the agent's role expands.

No red-teaming before production access

Teams test for the happy path but skip adversarial testing until something goes wrong in the wild.

How to avoid itRed-team agents against prompt injection, privilege escalation and data exfiltration before they touch production — and again after any significant change.

Guardrails applied only to user input, not tool output

It's intuitive to filter what the user types, but an agent's retrieved documents and tool responses are an equally real injection surface that often goes unguarded.

How to avoid itApply content filtering and validation to everything the agent reads, not just what the user sends.

Treating a single pilot's success as proof the architecture will scale

A narrow proof-of-concept often skips the governance, monitoring and cost controls a production-scale agentic system actually needs.

How to avoid itDesign the governance and guardrail layer alongside the pilot, not after it's approved for wider rollout.

Shadow AI usage with no visibility into what data leaves the organization

Staff adopt convenient public AI tools faster than policy and tooling can account for, often without realizing the data-handling implications.

How to avoid itOffer an approved, sanctioned AI tool that's genuinely as convenient as the shadow alternative, backed by a clear acceptable-use policy.

No audit trail of what an agent did or why

When an agent takes a consequential action, teams often can't reconstruct the reasoning or tool calls that led to it, making incidents hard to investigate and nearly impossible to prevent from recurring.

How to avoid itLog every tool call, input and decision an agent makes, retained long enough to support real incident investigation.

Guardrails tested once at launch, never revisited after a silent model upgrade

Vendors update underlying models without warning, and behavior that was safe under the old model isn't guaranteed to stay safe under the new one.

How to avoid itRe-run safety and guardrail tests on a schedule, and immediately after any detected model or provider change.

AI output used without disclosing AI involvement to the people affected by it

Teams ship AI-assisted decisions or content without flagging it, which erodes trust if discovered and increasingly runs into disclosure requirements.

How to avoid itDefault to disclosure wherever an AI system materially shapes what a user sees or decides, even where not yet strictly mandated.

AI Testing & Evaluations

How is testing an LLM application different from testing regular software?

Regular software testing checks deterministic pass/fail behavior; LLM applications produce probabilistic, varying outputs, so evaluation relies on quality metrics and scoring rubrics rather than simple equality checks.

Do we need a human to review every AI output before evaluation counts?

No. Most mature pipelines use automated metrics and LLM-as-a-judge scoring for scale, reserving human review for periodic spot-checks and safety-critical categories where automated scoring alone isn't trustworthy enough.

How often should we re-evaluate a model already in production?

Continuously, ideally. An automated evaluation pipeline should re-run whenever the model, prompts or underlying data change, since any of those can silently degrade quality without a visible error.

What's a reasonable hallucination rate to aim for?

There's no universal number — it depends heavily on use-case risk — but it should be measured against a defined baseline and tracked over time, not left unmeasured until a customer notices.

Can we trust an LLM to evaluate another LLM's output?

With care. LLM-as-a-judge scales evaluation well but carries its own biases and blind spots, so it's best combined with periodic human review, especially for safety or compliance-sensitive outputs.

Do we need a specialized evaluation platform, or can we build this ourselves?

Many organizations start with a lightweight internal pipeline using open-source evaluation libraries, then move to a dedicated platform once evaluation needs to run continuously across multiple models, prompts and release cycles.

How is evaluating an AI feature different from writing unit tests for it?

Unit tests check deterministic, known-correct outputs for fixed inputs. Evaluation scores quality across a representative distribution of inputs using rubrics and metrics, because the same prompt can legitimately produce different acceptable outputs each run — there often isn't one "correct" answer to assert against.

What metrics actually matter for a RAG system specifically?

Groundedness (does the answer stick to what the retrieved context actually says), retrieval relevance (did it fetch the right documents at all), and answer relevance to the question — measuring only final-answer quality without these hides whether failures come from retrieval or generation.

How do we build a golden dataset for evaluation?

Curate a representative set of real or realistic inputs — including edge cases and known-hard examples — paired with expert-approved expected outputs or scoring rubrics, then version it so every future evaluation run compares against the same fixed baseline.

What's a known failure mode of using LLM-as-a-judge?

Judge models can show systematic biases — favoring longer answers, being lenient toward outputs in a style similar to their own training, or missing subtle factual errors a domain expert would catch — so periodic human review stays necessary even in an LLM-judged pipeline.

How do we catch silent model drift after a provider updates their model?

Run your evaluation suite on a schedule against a fixed baseline, and flag any unexpected score movement — a quietly-upgraded underlying model can change behavior on your specific use case without any announcement from the vendor.

Do we need separate evaluation for safety versus quality?

Yes. Quality metrics (helpfulness, accuracy, relevance) and safety metrics (harmful content, policy violations, jailbreak resistance) can move independently, and optimizing purely for quality has been shown to sometimes quietly erode safety behavior.

Common challenges & how to avoid them

Evaluating only the happy path, missing edge cases

Test sets built from obvious examples miss the ambiguous, adversarial and out-of-scope queries that cause most real-world failures.

How to avoid itDeliberately build edge-case and adversarial test sets, not just representative "normal" examples.

No evaluation baseline to compare against

Without a recorded baseline, it's impossible to tell whether a new model or prompt version is actually better or worse.

How to avoid itEstablish and version a golden dataset and baseline scores before making changes, so every update is measured against something.

Evaluation stops after launch

Teams invest heavily in pre-launch testing, then treat evaluation as done — but models, prompts and real usage all drift over time.

How to avoid itWire evaluation into the CI/CD pipeline so it re-runs automatically on every relevant change.

Bias and fairness testing treated as optional

It's often skipped under time pressure since it doesn't block obvious functional failures.

How to avoid itMake demographic and phrasing-variation testing a required gate before launch, not a nice-to-have.

Metrics chosen because they're easy to compute, not because they measure what matters

Teams default to generic similarity or overlap scores that are simple to automate, which can diverge sharply from what actually makes an output good for the real use case.

How to avoid itPick metrics that map to the specific failure modes that matter for your product, even when they take more effort to compute.

No regression testing when switching model or provider versions

A prompt tuned carefully against one model version can behave noticeably differently on the next — teams often discover this only after users complain.

How to avoid itRe-run the full evaluation suite against any model or provider change before rolling it out, treating it like any other breaking dependency update.

Safety evaluation skipped for "low-risk" internal tools that later get exposed externally

An internal prototype built without safety rigor often gets fast-tracked to a customer-facing feature once it proves useful, carrying none of the scrutiny that promotion should have triggered.

How to avoid itApply the same evaluation bar before any internal tool is exposed externally, regardless of how it started.

Digital Testing & Automation

How much of our testing should be automated vs manual?

Automate the stable, repetitive checks — regression, smoke tests — to free up time for manual and exploratory testing on new, complex or high-risk features, where human judgment catches more than a script will.

What's a healthy test coverage percentage to aim for?

There's no universal target. High coverage on trivial code is less valuable than solid coverage on your highest-risk, most frequently changed logic — track coverage as a signal, not a goal in itself.

Why do our automated tests keep breaking on minor UI changes?

This usually means tests are tied too tightly to brittle locators, like exact pixel positions or auto-generated IDs. More resilient locator strategies, or self-healing test automation, reduce this maintenance burden significantly.

Should every release go through full performance testing?

No. Scale the depth of non-functional testing to the risk of the release — a small config change doesn't need a full load test, but anything touching authentication, payments or core infrastructure should.

What's the actual ROI of test automation?

It shows up over time, not immediately. Automation has an upfront build cost, but pays off through faster regression cycles, fewer production defects, and freeing testers for exploratory and high-value testing.

How do we handle testing across so many device and browser combinations?

Prioritize by actual usage data from your analytics rather than testing every theoretical combination, and use cloud-based device/browser testing grids to cover the long tail efficiently.

What's the difference between shift-left and shift-right testing?

Shift-left moves testing earlier — into design and development, before code reaches QA — to catch defects cheaply. Shift-right tests in or close to production — canary releases, synthetic monitoring, feature flags — to catch what only shows up under real traffic and data. Mature programmes do both, not one instead of the other.

Should developers write their own tests, or should that be QA's job exclusively?

Developers should own unit and component-level tests close to their code; QA should own integration, end-to-end and exploratory testing where a broader, user-facing view matters more. Splitting it this way catches more than either group working in isolation.

What's contract testing and when do we actually need it?

Contract testing verifies that a service's API still meets the expectations its consumers rely on, independent of a full end-to-end test. It becomes valuable once you have multiple services or teams integrating via APIs — it catches breaking changes at the source, before they cascade downstream.

Can self-healing test automation actually be trusted?

It meaningfully reduces maintenance from routine UI changes by automatically adapting locators, but it isn't infallible — it can occasionally "heal" a test past what should have been a genuine failure, so treat its changes as something to spot-check periodically, not something to fully automate out of review.

What's a reasonable split between unit, integration and end-to-end tests?

The classic "testing pyramid" — many fast unit tests, fewer integration tests, a lean set of end-to-end tests — still holds up well: end-to-end tests are the most realistic but also the slowest and most brittle, so they should cover critical paths, not every scenario.

Does visual regression testing need to be automated, or is manual review enough?

For any product with frequent UI changes, automated visual regression (pixel/DOM diffing against a baseline) catches unintended changes far more reliably and cheaply than manual eyeballing, especially across many screen sizes and themes.

Common challenges & how to avoid them

Flaky tests eroding trust in the test suite

Tests that fail intermittently for reasons unrelated to real defects train teams to ignore failures altogether.

How to avoid itTrack and fix flaky tests as a priority — don't just rerun until green.

Automation written to match existing manual scripts, not redesigned for automation

Converting manual test scripts one-to-one often produces brittle, hard-to-maintain automated tests.

How to avoid itDesign automated tests around stable, maintainable patterns — not a literal translation of manual steps.

Test environments that don't match production

Bugs that never surface in staging can appear immediately in production due to data, scale or configuration differences.

How to avoid itKeep staging as close to production as practically possible, and test some scenarios against production-like data volumes.

No clear owner for test data management

Stale, unrealistic or inconsistent test data produces unreliable test results and false confidence.

How to avoid itTreat test data as a maintained asset with its own owner and refresh process, not an afterthought.

Test suite ownership unclear between QA and developers

When no one is explicitly responsible for a test, it slowly rots — nobody fixes it when it breaks, and eventually someone just disables it.

How to avoid itAssign explicit ownership for every test suite, and make broken-test triage part of the regular workflow, not a backlog item.

CI pipeline so slow that teams start skipping tests to ship faster

A test suite that takes too long to run creates pressure to bypass it under deadline, defeating its purpose right when it matters most.

How to avoid itParallelize test execution and prune or optimize the slowest tests — a fast suite gets run; a slow one gets skipped.

Accessibility and performance testing left entirely out of the automated suite

These checks are treated as specialist, separate activities and so never run as part of everyday regression testing, letting regressions ship silently.

How to avoid itFold automated accessibility and performance checks into the same CI pipeline as functional tests, not a separate occasional audit.

Test reports that nobody actually reads

A dashboard full of pass/fail results that no one reviews might as well not exist — failures sit unaddressed for weeks.

How to avoid itRoute test failures to the people who can act on them immediately, not into a report that requires someone to go looking.

Strategic IT & Architecture Advisory

When does it make sense to move from a monolith to microservices?

When a monolith's size is genuinely slowing teams down — deployment bottlenecks, being forced to scale the whole system to scale one part, team coordination overhead — not simply because microservices are currently fashionable. The added operational complexity isn't free.

How do we justify an architecture modernization budget to leadership?

Frame it around the business cost of not modernizing — rising maintenance cost, slower time-to-market, security and compliance risk, and the growing difficulty of hiring for an aging stack — rather than purely technical merits.

Do we need a full enterprise architecture practice, or can we start smaller?

You can start smaller. Even a lightweight current-state/target-state exercise for your highest-priority domain delivers real value before investing in a full EA practice and tooling.

Is multi-cloud worth the added complexity?

Only if it's solving a specific need — resilience, a regulatory requirement, or avoiding vendor lock-in for a critical capability. Adopted without a clear reason, multi-cloud usually just adds operational cost without a matching benefit.

How do we keep cloud costs under control as we scale?

Give engineering and product teams visibility into what their own usage actually costs — a FinOps practice — rather than leaving cost management solely to finance after the bill arrives.

What's the biggest risk in a legacy modernization project?

Underestimating the undocumented business logic buried in the legacy system. "Big bang" rewrites that don't account for it tend to fail; an incremental, strangler-pattern approach that migrates piece by piece is usually lower-risk.

What's the difference between enterprise architecture and solution architecture?

Enterprise architecture looks across the whole organization — how business capabilities, data and systems fit together strategically. Solution architecture designs a single system or project within that broader context. Most organizations need solution-level discipline well before they need a dedicated EA practice.

How do we decide between building a capability ourselves or buying it?

Build when the capability is close to your actual competitive differentiation and you need control over its roadmap; buy when it's a commodity function (auth, payments, email) where a mature vendor will out-invest what you could justify building and maintaining yourselves.

Do we need to follow a formal framework like TOGAF, or can we work more informally?

Most organizations benefit from TOGAF's concepts — current-state/target-state mapping, architecture decision records, governance gates — without adopting the full formal process. Treat it as a toolbox to borrow from, not a certification to chase for its own sake.

How should AI change our architecture roadmap?

Plan for AI agents as a new category of system consumer — they need their own authentication, rate limits, auditable action boundaries and monitoring, distinct from how you've architected for human users and traditional service-to-service calls.

What's the difference between a platform engineering team and a DevOps team?

DevOps is a practice — shared responsibility for build, deploy and operate across dev and ops. Platform engineering is a dedicated team that builds internal tooling and self-service infrastructure so other teams can practice DevOps effectively without each reinventing it.

How do we know when to replace a legacy system versus keep modernizing it incrementally?

Replace when the platform itself is the constraint — it can't be scaled, secured or integrated no matter how much you invest. Modernize incrementally when the core platform is sound but specific components or practices around it have fallen behind.

Common challenges & how to avoid them

Architecture decisions made without a documented rationale

Decisions feel obvious at the time, but the reasoning is lost as teams and priorities change, leading to repeated debates and inconsistent choices later.

How to avoid itRecord key architecture decisions and their rationale — an Architecture Decision Record — as you make them.

Technical debt that's invisible until it blocks a roadmap

Debt accumulates gradually and competes poorly against visible feature work for budget and attention.

How to avoid itTrack technical debt explicitly and give it a defined share of every planning cycle, not just "whenever there's time."

Platform engineering initiatives built without understanding what developers actually need

An internal platform designed in isolation from the teams who'll use it often goes unadopted, no matter how well-engineered.

How to avoid itTreat the platform as a product with real developer users, and validate it with them continuously.

Vendor lock-in discovered only when trying to leave

Convenience features and proprietary APIs get adopted incrementally without anyone weighing the exit cost.

How to avoid itFactor portability and exit cost into vendor and platform decisions upfront, not after you're already committed.

Architecture review board seen as a bottleneck and routinely bypassed

When governance feels slow and disconnected from delivery pressure, teams route around it, and architecture integrity erodes one exception at a time.

How to avoid itMake the review process fast and lightweight for low-risk changes, reserving real scrutiny for decisions that actually warrant it.

No single source of truth for current-state architecture

Every team maintains its own mental model or outdated diagram, so decisions get made on inconsistent assumptions about what actually exists.

How to avoid itMaintain one living, accessible architecture record, and make keeping it current part of how changes get shipped.

Cloud migration done as a "lift and shift" that locks in the same inefficiencies

Moving an unchanged application to the cloud carries over its on-prem assumptions and inefficiencies, often at a higher ongoing cost than before.

How to avoid itTreat migration as a chance to rearchitect the highest-cost, highest-friction components, not just relocate them unchanged.

Integration sprawl — point-to-point connections multiplying without a strategy

Each new integration gets built the fastest available way, and the resulting web of direct connections becomes unmaintainable and fragile to change.

How to avoid itAdopt a deliberate integration pattern — an API gateway or event bus — before the number of point-to-point connections gets out of hand.

Still not sure where to start?

A short conversation usually answers more than a page can — tell us what you're working on.