Post by habiba123820 on Nov 6, 2024 1:42:50 GMT -4
We define a large-scale connector strategy as one that produces more than 5000 Units of Work in BWX per sync request. A Unit of Work in BWX is a combination of a file + workflow step + language pair. So a project with 5 files, 2 workflow steps (Translation + Review) and 5 languages will result in 50 Units of Work. You can see how it is not that difficult to exceed 5000 Units of Work when you start multiplying these variables. It is tempting to want to centralize everything into a single project, but our experience shows that with large-scale connectors, grouping projects by language pair is the best way to proceed. This may seem like a simple matter of how you choose to define a project, but it goes much deeper than that. In this article, we will analyze the impact of this decision in terms of:
Message Brokering1) Messaging grows exponentially with large scale connectors. We have seen connectors that create over 1MM messages per sync request. This results in tremendous server activity and performance issues that can be mitigated by splitting projects by locale. Troubleshooting2) This is directly related to risk mitigation. But in localization, we often encounter issues that are restricted to a particular region and how that affects the parsing, segmentation, and
pre/post processing of a given set of files. By splitting across locales, you can create locale-specific Regular Expressions, processing rules, segmentation that give you much more flexibility when it wordpress web design agency comes to the overall architecture. Instead of working with one-size-fits-all fixes, you can iterate based on locale and eventually achieve a more mature and predictable pattern of behavior when working across locales. Risk Mitigation 3) By decoupling projects into one per locale, you mitigate management risk because if something goes wrong in a given locale, it doesn’t mean that the entire pull request/delivery mechanism is compromised. You can isolate and naturally compartmentalize problems.
This may not seem like a big deal during the SOP, but when unexpected problems arise (and they always do in localization), you’ll be glad you built a house of bricks instead of one made of straw. Queuing/Performance 4) Instead of queuing 150,000 elements for processing, for example, you can queue 15,000 elements ten times. Again, this doesn’t seem like a huge difference since you’ll end up processing the same 150,000 elements, but having the flexibility to process serially vs. in parallel or opportunistically as desired gives you much more flexibility as well as performance bandwidth. Automation Potential 5) Typically, project decisions and workflows will be asymmetric across locales. Separating projects by location gives you much greater flexibility in terms of long-term automation potential. You may have a scenario where you have completely different parameters and data sets when you segregate by location, as opposed to consolidating everything together. Management6) This is also counterintuitive.
From a management perspective, consolidation is generally the best practice for better governance. But with large-scale connectors, the opposite is true. A project will naturally filter by location, allowing different project managers to own different parts of the project more easily, reducing the use of filters for reporting, and creating greater simplicity in tracking what is happening by location. Scalability7) With large-scale connectors,You will reach a point where it simply becomes impossible to scale by bundling all the work units into a single project. By breaking things apart, you establish the framework for a program that is easier to scale in the long run. Remember, things multiply when it comes to work units and messages.
By breaking things out by location, you eliminate one of the big multipliers, allowing you to scale much more easily. Bottom line: There is an illusion that consolidation is better. One pull request per project makes everyone's life easier. While this is true for small-scale connectors, it falls apart for large-scale connectors. Our goal is to always deliver the most elegant and reliable solutions to our customers, and we have seen time and time again that in large-scale situations, divide and conquer is the way to go.
Consolidating into a single projectSeparating by localeEntire project has a status of 0 or 1Status can be stratified by localeSingular parsing, filtering and RegEx frameworkFlexible framework by localeOne-way queuing by projectFlexible queuing and distributed processing naturallyFiltering by locale within the projectOne less filtering stepAutomation rules that work in all casesLocale-specific automation parameters and dataAll eggs in one basket from a risk perspectiveRisk distributed across localesComplex to isolate and troubleshootOne less factor of large variability making it easier to troubleshoot
Message Brokering1) Messaging grows exponentially with large scale connectors. We have seen connectors that create over 1MM messages per sync request. This results in tremendous server activity and performance issues that can be mitigated by splitting projects by locale. Troubleshooting2) This is directly related to risk mitigation. But in localization, we often encounter issues that are restricted to a particular region and how that affects the parsing, segmentation, and
pre/post processing of a given set of files. By splitting across locales, you can create locale-specific Regular Expressions, processing rules, segmentation that give you much more flexibility when it wordpress web design agency comes to the overall architecture. Instead of working with one-size-fits-all fixes, you can iterate based on locale and eventually achieve a more mature and predictable pattern of behavior when working across locales. Risk Mitigation 3) By decoupling projects into one per locale, you mitigate management risk because if something goes wrong in a given locale, it doesn’t mean that the entire pull request/delivery mechanism is compromised. You can isolate and naturally compartmentalize problems.
This may not seem like a big deal during the SOP, but when unexpected problems arise (and they always do in localization), you’ll be glad you built a house of bricks instead of one made of straw. Queuing/Performance 4) Instead of queuing 150,000 elements for processing, for example, you can queue 15,000 elements ten times. Again, this doesn’t seem like a huge difference since you’ll end up processing the same 150,000 elements, but having the flexibility to process serially vs. in parallel or opportunistically as desired gives you much more flexibility as well as performance bandwidth. Automation Potential 5) Typically, project decisions and workflows will be asymmetric across locales. Separating projects by location gives you much greater flexibility in terms of long-term automation potential. You may have a scenario where you have completely different parameters and data sets when you segregate by location, as opposed to consolidating everything together. Management6) This is also counterintuitive.
From a management perspective, consolidation is generally the best practice for better governance. But with large-scale connectors, the opposite is true. A project will naturally filter by location, allowing different project managers to own different parts of the project more easily, reducing the use of filters for reporting, and creating greater simplicity in tracking what is happening by location. Scalability7) With large-scale connectors,You will reach a point where it simply becomes impossible to scale by bundling all the work units into a single project. By breaking things apart, you establish the framework for a program that is easier to scale in the long run. Remember, things multiply when it comes to work units and messages.
By breaking things out by location, you eliminate one of the big multipliers, allowing you to scale much more easily. Bottom line: There is an illusion that consolidation is better. One pull request per project makes everyone's life easier. While this is true for small-scale connectors, it falls apart for large-scale connectors. Our goal is to always deliver the most elegant and reliable solutions to our customers, and we have seen time and time again that in large-scale situations, divide and conquer is the way to go.
Consolidating into a single projectSeparating by localeEntire project has a status of 0 or 1Status can be stratified by localeSingular parsing, filtering and RegEx frameworkFlexible framework by localeOne-way queuing by projectFlexible queuing and distributed processing naturallyFiltering by locale within the projectOne less filtering stepAutomation rules that work in all casesLocale-specific automation parameters and dataAll eggs in one basket from a risk perspectiveRisk distributed across localesComplex to isolate and troubleshootOne less factor of large variability making it easier to troubleshoot