[ad_1]
It is a visitor put up co-authored with Kanehito Miyake, Engineer at Fujitsu Japan.
Fujitsu Restricted was established in Japan in 1935. At present, now we have roughly 120,000 staff worldwide (as of March 2023), together with group firms. We develop enterprise in varied areas around the globe, beginning with Japan, and supply digital companies globally. To supply quite a lot of merchandise, companies, and options which can be higher suited to clients and society in every area, now we have constructed enterprise processes and techniques which can be optimized for every area and its market.
Nonetheless, lately, the IT market setting has modified drastically, and it has change into tough for your entire group to reply flexibly to the person market state of affairs. Furthermore, we’re challenged not solely to revisit particular person merchandise, companies, and options, but additionally to reinvent complete enterprise processes and operations.
To rework Fujitsu from an IT firm to a digital transformation (DX) firm, and to change into a world-leading DX associate, Fujitsu has declared a shift to data-driven administration. We constructed the OneFujitsu program, which standardizes enterprise initiatives and techniques all through the corporate, together with the home and abroad group firms, and tackles the foremost transformation of your entire firm beneath this system.
To attain data-driven administration, we constructed OneData, a knowledge utilization platform used within the 4 world AWS Areas, which began operation in April 2022. As of November 2023, greater than 200 initiatives and 37,000 customers have been onboarded. The platform consists of roughly 370 dashboards, 360 tables registered within the information catalog, and 40 linked techniques. The information measurement saved in Amazon Easy Storage Service (Amazon S3) exceeds 100 TB, together with information processed to be used in every venture.
On this put up, we introduce our OneData initiative. We clarify how Fujitsu labored to resolve the aforementioned points and introduce an summary of the OneData design idea and its implementation. We hope this put up will present some steerage for architects and engineers.
Challenges
Like many different firms battling information utilization, Fujitsu confronted some challenges, which we focus on on this part.
Siloed information
In Fujitsu’s lengthy historical past, we restructured organizations by merging affiliated firms into Fujitsu. Though organizational integration has progressed, there are nonetheless many techniques and mechanisms personalized for particular person context. There are additionally many techniques and mechanisms overlapping throughout completely different organizations. Because of this, it takes numerous effort and time to find, search, and combine information when analyzing your entire firm utilizing a typical customary. This example makes it tough for administration to know enterprise tendencies and make choices in a well timed method.
Below these circumstances, the OneFujitsu program is designed have one system per one enterprise globally. Core techniques comparable to ERP and CRM are being built-in and unified so as to not have silos. It should make it simpler for customers to make the most of information throughout completely different organizations for particular enterprise areas.
Nonetheless, to unfold a tradition of data-driven decision-making not solely in administration but additionally in each group, it’s essential to have a mechanism that allows customers to simply uncover varied kinds of information in organizations, after which analyze the info rapidly and flexibly when wanted.
Excel-based information utilization
Microsoft Excel is accessible on nearly everybody’s PC within the firm, and it helps decrease the hurdles when beginning to make the most of information. Nonetheless, Excel is especially designed for spreadsheets; it’s not designed for large-scale information analytics and automation. Excel recordsdata are inclined to comprise a combination of information and procedures (features, macros), and lots of customers casually copy recordsdata for one-time use instances. It introduces complexity to maintain each information and procedures updated. Moreover, it tends to require domain-specific information to handle the Excel recordsdata for particular person context.
For these causes, it was extraordinarily tough for Fujitsu to handle and make the most of information at scale with Excel.
Resolution overview
OneData defines three personas:
- Writer – This function contains the organizational and administration crew of techniques that function information sources. Tasks embody:
- Load uncooked information from the info supply system on the acceptable frequency.
- Present and maintain updated with technical metadata for loaded information.
- Carry out the cleaning course of and format conversion of uncooked information as wanted.
- Grant entry permissions to information based mostly on the requests from information customers.
- Client – Shoppers are organizations and initiatives that use the info. Tasks embody:
- Search for the info for use from the technical information catalog and request entry to the info.
- Deal with the method and conversion of information right into a format appropriate for their very own use (comparable to fact-dimension) with granted referencing permissions.
- Configure enterprise intelligence (BI) dashboards to offer data-driven insights to end-users focused by the buyer’s venture.
- Use the newest information printed by the writer to replace information as wanted.
- Promote and increase using databases.
- Basis – This function encompasses the info steward and governance crew. Tasks embody:
- Present a preprocessed, generic dataset of information generally utilized by many customers.
- Handle and information metrics for the standard of information printed by every writer.
Every function has sub-roles. For instance, the buyer function has the next sub-roles with completely different tasks:
- Information engineer – Create information course of for evaluation
- Dashboard developer – Create a BI dashboard
- Dashboard viewer – Monitor the BI dashboard
The next diagram describes how OneData platform works with these roles.
Let’s take a look at the important thing elements of this structure in additional element.
Writer and shopper
Within the OneData platform, the writer is per every information supply system, and the buyer is outlined per every information utilization venture. OneData offers an AWS account for every.
This allows the writer to cleanse information and the buyer to course of and analyze information at scale. As well as, by correctly separating information and processing, it turns into easy for the groups and organizations to share, handle, and inherit processes that have been historically confined to particular person PCs.
Basis
When the groups don’t have a strong sufficient skillset, it could actually require extra time to mannequin and course of information, and trigger longer latency and decrease information high quality. It may well additionally contribute to decrease utilization by end-users. To handle this, the muse function offers an already processed dataset as a generic information mannequin for information generally use instances utilized by many customers. This allows high-quality information obtainable to every shopper. Right here, the muse function takes the lead in compiling the information of area specialists and making information appropriate for evaluation. Additionally it is an efficient strategy that eliminates duplicates for customers. As well as, the muse function screens the state of the metadata, information high quality indicators, information permissions, data classification labels, and so forth. It’s essential in information governance and information administration.
BI and visualization
Particular person customers have a devoted area in a BI device. Up to now, if customers needed to transcend easy information visualization utilizing Excel, they needed to construct and preserve their very own BI instruments, which precipitated silos. By unifying these BI instruments, OneData lowers the issue for customers to make use of BI instruments, and centralizes operation and upkeep, attaining optimization on a company-wide scale.
Moreover, to maintain portability between BI instruments, OneData recommends customers rework information inside the shopper AWS account as a substitute of reworking information within the BI device. With this strategy, BI device masses information from AWS Glue Information Catalog tables by an Amazon Athena JDBC/ODBC driver with none additional transformations.
Deployment and operational excellence
To supply OneData as a typical service for Fujitsu and group firms around the globe, Regional OneData has been deployed in a number of places. Regional OneData represents a unit of system configurations, and is designed to offer decrease community latency for platform customers, and be optimized for native languages, working hours for system operations and assist, and region-specific authorized restrictions, comparable to information residency and private data safety.
The Regional Operations Unit (ROU), a digital group that brings collectively members from every area, is accountable for working regional OneData in every of those areas. OneData HQ is accountable for supervising these ROUs, in addition to planning and managing your entire OneData.
As well as, now we have a specifically positioned OneData referred to as International OneData, the place world information utilization spans every area. Solely the correctly cleansed and sanitized information is transferred between every Regional OneData and International OneData.
Techniques comparable to ERP and CRM are accumulating information as a writer for International OneData, and the dashboards for executives in varied areas to watch enterprise circumstances with world metrics are additionally appearing as a shopper for International OneData.
Technical ideas
On this part, we focus on among the technical ideas of the answer.
Massive scale multi-account
Now we have adopted a multi-account technique to offer AWS accounts for every venture. Many publishers and customers are already onboarded into OneData, and the quantity is anticipated to extend sooner or later. With this technique, future utilization enlargement at scale might be achieved with out affecting the customers.
Additionally, this technique allowed us to have clear boundaries in safety, prices, and repair quotas for every AWS service.
All of the AWS accounts are deployed and managed by AWS Organizations and AWS Management Tower.
Serverless
Though we offer impartial AWS accounts for every writer and shopper, each operational prices and useful resource prices can be monumental if we accommodated particular person person requests, comparable to, “I need a digital machine or RDBMS to run particular instruments for information processing.” To keep away from such steady operational and useful resource prices, now we have adopted AWS serverless companies for all of the computing sources needed for our actions as a writer and shopper.
We use AWS Glue to preprocess, cleanse, and enrich information. Optionally, AWS Lambda or Amazon Elastic Container Service (Amazon ECS) with AWS Fargate may also be used based mostly on preferences. We enable customers to arrange AWS Step Features for orchestration and Amazon CloudWatch for monitoring. As well as, we offer Amazon Aurora Serverless PostgreSQL as customary for customers, to satisfy their wants for information processing with extract, load, and rework (ELT) jobs. With this strategy, solely the buyer who requires these companies will incur expenses based mostly on utilization. We’re in a position to reap the benefits of decrease operational and useful resource prices because of the distinctive advantage of serverless (or extra precisely, pay-as-you-go) companies.
AWS offers many serverless companies, and OneData has built-in them to offer scalability that enables lively customers to rapidly present the required functionality as wanted, whereas minimizing the fee for non-frequent customers.
Information possession and entry management
In OneData, now we have adopted a knowledge mesh structure the place every writer maintains possession of information in a distributed and decentralized method. When the buyer discovers the info they need to use, they request entry from the writer. The writer accepts the request and grants permissions solely when the request meets their very own standards. With the AWS Glue Information Catalog and AWS Lake Formation, there isn’t a have to replace S3 bucket insurance policies or AWS Id and Entry Administration (IAM) insurance policies each time we enable entry for particular person information on an S3 information lake, and we are able to effortlessly grant the mandatory permissions for the databases, tables, columns, and rows when wanted.
Conclusion
Because the launch of OneData in April 2022, now we have been persistently finishing up academic actions to increase the variety of customers and introducing success tales on our portal website. Consequently, now we have been selling change administration inside the firm and are actively using information in every division. Regional OneData is being rolled out regularly, and we plan to additional increase the dimensions of use sooner or later.
With its world enlargement, the event of fundamental features as a knowledge utilization platform will attain a milestone. As we transfer ahead, it will likely be necessary to guarantee that OneData platform is used successfully all through Fujitsu, whereas incorporating new applied sciences associated to information evaluation as acceptable. For instance, we’re making ready to offer extra superior machine studying features utilizing Amazon SageMaker Studio with OneData customers and investigating the applicability of AWS Glue Information High quality to cut back the guide high quality monitoring efforts. Moreover, we’re presently within the means of implementing Amazon DataZone by varied initiatives and efforts, comparable to verifying its performance and analyzing the way it can function whereas bridging the hole between OneData’s current processes and to the perfect course of we’re aiming for beliefs.
Now we have had the chance to debate information utilization with varied companions and clients and though particular person challenges could differ in measurement and its context, the problems that we’re presently making an attempt to resolve with OneData are frequent to lots of them.
This put up describes solely a small portion of how Fujitsu tackled challenges utilizing the AWS Cloud, however we hope the put up will provide you with some inspiration to resolve your personal challenges.
In regards to the Writer
Kanehito Miyake is an engineer at Fujitsu Japan and in command of OneData’s answer and cloud structure. He spearheaded the architectural examine of the OneData venture and contributed drastically to selling information utilization at Fujitsu along with his experience. He loves rockfish fishing.
Junpei Ozono is a Go-to-market Information & AI options architect at AWS in Japan. Junpei helps clients’ journeys on the AWS Cloud from Information & AI elements and guides them to design and develop data-driven architectures powered by AWS companies.
[ad_2]